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Foreword 


In 2010 the organisations that form the E4 Group, notably ENQA, ESU, EUA and 
EUR ASHE launched the “Mapping the Implementation and Application of the 
Standards and Guidelines for Quality Assurance in the European Higher Education 
Area (MAP-ESG)” project in order to gather information on how the ESG have been 
implemented and applied in the 47 Bologna signatory countries in higher education 
institutions and in quality assurance institutions, and to launch a thorough discussion 
on the ESG in their entirety. In doing so the E4 Group responded to a discussion which 
has gained momentum since 2009 and which suggested that there should be a review 
of the ESG. 

This report presents the results of the activities carried out in the context of the 
MAP-ESG project. In doing so the report provides the first overarching study on the 
implementation of the ESG in all its parts, and a thorough analysis of their impact. 

The report is based on the analysis of expert views from all relevant actors and 
stakeholders in higher education, notably teachers and students, higher education 
management and internal quality assurance bodies, external quality assurance 
agencies, EQAR and ministries, and the social partners. 

Therefore the recommendation of the project consortium to the Bologna Follow-up 
Group and the Ministerial meeting in Bucharest in April 2012 is to mandate a careful 
revision of the ESG in order to improve their clarity, applicability and usefulness. 

It integrates the perspectives of all stakeholders and, thus, forms a sound basis for 
ministers to take a well-informed decision on the further development of the ESG. 

On behalf of the E4 Group I would like to thank the contributors to this project. 
These are the many experts who took part in the consultations, the members of the 
steering group and also the members of the advisory board. 

I do hope that this report facilitates a further development of the ESG which have 
been a success story since they were adopted within the Bologna Process. I hope that 
the report leads to the enhancement of the applicability of the ESG and further fosters 
their implementation. Since they are already a success story, the ESG would thus be 
able to make an even more effective contribution to quality assurance in the EHEA. 

Achim Hopbach 

President of ENQA 
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Executive Summary 


Introduction 

Since the adoption of the Standards and Guidelines for Quality Assurance in the EHEA 
(ESG) by ministers in Bergen (2005), considerable progress has been made in their 
implementation and impact. However, the 2005 report on the ESG stated that they 
should not be considered as unchangeable (ENQA 2005: 13); it was always clear to the 
authors that once some time for implementation had elapsed, the ESG would need to be 
analysed and reviewed. 

The European quality assurance landscape has changed considerably in recent 
years. This has already been acknowledged by the E4 organisations in their “Report 
to the London conference of Ministers on a European Register of Quality Assurance 
Agencies” in 2007; and in 2009 the European Commission published a “Report on 
progress in quality assurance in higher education” which included a recommendation 
to revisit the ESG. (EC 2009: 5) Until now, there has been no overarching study on the 
implementation of the ESG in all its parts, and no thorough analysis of the impact of 
the ESG had yet been carried out. 

With this background in mind, the “Mapping the Implementation and Application 
of the ESG (MAP-ESG)” project was launched by the E4 Group (ENQA, ESU, EUA and 
EURASHE) in order to gather information on how the ESG have been implemented 
and applied in the 47 Bologna signatory countries in HEIs and in QAAs and to launch a 
thorough discussion on the ESG in their entirety. 

A Steering Group 1 representative of each of the E4 Group consulted their respective 
membership, and drew conclusions and recommendations from those consultations, 
which then fed into this joint report to the Ministers. The report is complemented by 
the individual reports of the four organisations as annexes. In order to ensure wider 
involvement of other stakeholders, the project was assisted by an Advisory Board, 
consisting of EQAR, Business Europe, Education International and government 
representatives identified by the BFUG. 

Structure of the report 

The report is structured into five chapters plus the annexes. The chapters are: 

Chapter 1: An introduction to the ESG, their background and history and the 
discussion that led to the review carried out through the MAP-ESG project 

Chapter 2: An introduction to the MAP-ESG project itself, the project partners and 
the methodology used to reach the final report 

Chapter 3: The major findings of the project, divided into three sub-headings 
(the Purpose and Scope of the ESG, the Clarity and Usability of the ESG and the 
Implementation and Impact of the ESG) 

Chapter 4: Conclusions of the project (under the same three sub-headings) 

Chapter 5: The recommendation to ministers 

Annexes (the four reports from the individual consultations of the project partners, 
list of abbreviations and list of the Steering Group members) 


1 See Annex 6 for a full list of the Steering Group members. 
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Findings and conclusions of the project 

As is clearly evidenced by the findings of the MAP-ESG project, the ESG have proved 
to be a major achievement of the Bologna Process - they are well-regarded by all 
stakeholders, have proved to be applicable across diverse contexts, have impacted 
on the development of QA processes at institutional and national level and on the 
work and review of quality assurance agencies (QAAs). They have facilitated a shared 
understanding of QA amongst the relevant stakeholders and actors in HE. QA, whether 
internal or external, is conducted across the EHEA according to the framework of 
principles they provide. They constitute a crucial means for achieving the goals of the 
Bologna Process. 

THE FOLLOWING CONCLUSIONS WERE REACHED: 

Purpose and Scope: it was generally concluded that it is essential to maintain the 
concept of the generic principle in order to ensure the continuing relevance of the 
ESG to all relevant stakeholders in the EHEA and to maintain their authority as the 
common reference point for QA in the EHEA. A tension regarding the purpose of 
the ESG was raised: that of their role as a reference document as opposed to that of a 
compliance tool. The current scope is deemed to be generally appropriate but there is 
encouragement to reflect on the extent to which a revised ESG document should link 
to specific Bologna commitments and reflect overarching principles agreed among the 
Bologna signatories. 

Clarity and Usability: it is clear that the document could be improved by some 
further work to increase clarity of terminology and to ensure the removal of ambiguity 
both in terms of the language used and also with regard to the standards and guidelines 
themselves to ensure that they are as clear as possible. 

Implementation and Impact: there is clear evidence that the ESG have been 
widely implemented and have impacted significantly on QA in the EHEA. However, 
further work could be done to raise awareness and ownership of the ESG even further, 
particularly amongst faculty staff directly involved in the student learning and teaching 
process. 

The conclusions of the project, therefore, focus on the improvement of the ESG as 
they are currently formulated rather than recommending a wholesale revision of the 
content of the principles enshrined in the document. 

THE RECOMMENDATION TO MINISTERS 

Based on the findings of the project, “Mapping the implementation and application 
of the ESG” and the conclusions derived from the findings, ENQA, ESU, EUA and 
EURASHE recommend that ministers of the EHEA countries mandate the E4 organisations 
to carry out , in consultation with all relevant stakeholders , notably Education 
International Business Europe , the Bologna Follow Up Group (BFUG) and EQAR, a careful 
revision of the ESG in order to improve their clarity , applicability and usefulness. This work 
would be carried out in the understanding that the current principles would be maintained. 
The report , with the revised document , would be referred to the BFUG. 
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1. Introduction to the Standards and 
Guidelines for Quality Assurance in the 
European Higher Education Area (ESG): 
background and history 

One of the key aims of the Bologna Process is to improve the quality of European 
higher education as a means to promote its attractiveness and competitiveness as well 
as to facilitate mobility. While quality assurance (QA) is only cursorily mentioned in 
the first Ministerial Communiques, since 2003 it has been considered to be one of the 
key action lines in the Bologna Process and the European quality assurance framework 
has been a significant issue in Ministerial meetings. It has also proved to be one of the 
most successful action lines as can be evidenced by developments such as increasing E4 
cooperation, the significance of the European Quality Assurance Forum (EQAF) as a 
means of helping to promote the European dimension of QA and, indeed, the success of 
the ESG themselves. 

Developing the ESG 

In 2003, the Berlin Communique stated that “consistent with the principle of 
institutional autonomy, the primary responsibility for quality assurance in higher 
education lies with each institution itself”. The Ministers further called upon “ENQA 
through its members, in co-operation with the EUA, EURASHE and ESIB 2 , to develop 
an agreed set of standards, procedures and guidelines on quality assurance, to 
explore ways of ensuring an adequate peer review system for quality assurance and / 
or accreditation agencies or bodies, and to report back through the Follow-up Group to 
Ministers in 2005.” (Bologna Process Ministerial Communique (BPMC) 2003) 

This launched an intensive period of activity for the four stakeholder organisations 
(ENQA, ESIB/ESU, EUA and EURASHE). An increasingly structured cooperation 
process was established leading to the proposal to Ministers for European Standards 
and Guidelines for QA in the EHEA that took account of the perspective of quality 
assurance agencies (QAAs), higher education institutions (HEIs) and students. Since 
then the four organisations have continued to meet regularly to discuss matters related 
to quality assurance in a framework that has come to be known as the “E4 Group”. 

In 2005 Ministers of Higher Education meeting in Bergen adopted the Standards 
and Guidelines for Quality Assurance in the European Higher Education Area (ESG) 
based on the proposal prepared by the E4 Group. (BPMC 2005) 

The purpose and the nature of the ESG 

One of the most notable features of the ESG is that they were “designed to be applicable 
to all HEIs and QAAs in Europe, irrespective of their structure, function and size, and 
the national system in which they are located.” (ENQA, 2009, 12) 


2 ENQA: the European Association for Quality Assurance in Higher Education; EUA: the European University Association; 
EURASHE: the European Association of Institutions in Higher Education and ESIB, nowadays ESU: the European Students’ 
Union. 
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The ESG are divided into three parts: 

1. Internal quality assurance of higher education institutions (seven standards) 

2. external quality assurance of higher education (eight standards) 

3. quality assurance of external quality assurance agencies (eight standards) 

Each standard is followed by guidelines. In the E4 report (the original report in which 
the ESG were published) the relationship between the standards and guidelines is 
presented as follows: 

“The standards that follow reflect basic good practice across Europe in external 
quality assurance, but do not attempt to provide detailed guidance about what 
should be examined or how quality assurance activities should be conducted. 

[...] The guidelines’ provide additional information about good practice and in 
some cases explain in more detail the meaning and importance of the standards. 
Although the guidelines are not part of the standards themselves, the standards 
should be considered in conjunction with them.” (ENQA 2005: 15) 

The ESG state that the “purpose of these standards and guidelines is to provide a source 
of assistance and guidance both to higher education institutions in developing their 
own quality assurance systems and agencies undertaking external quality assurance, as 
well as to contribute to a common frame of reference, which can be used by institutions 
and agencies alike. It is not the intention that these standards and guidelines should 
dictate practice or be interpreted as prescriptive or unchangeable.” (ENQA 2005: 13) 

The document further defines the purpose of the standards and guidelines as being: 

• “to improve the education available to students in higher education institutions in 
the EHEA; 

• to assist higher education institutions in managing and enhancing their quality 
and, thereby, to help to justify their institutional autonomy; 

• to form a background for quality assurance agencies in their work; 

• to make external quality assurance more transparent and simpler to understand 
for everybody involved.” (ENQA 2005: 14) 

The objectives of the ESG are defined as being: 

• “to encourage the development of higher education institutions which foster 
vibrant intellectual and educational achievement; 

• to provide a source of assistance and guidance to higher education institutions and 
other relevant agencies in developing their own culture of quality assurance; 

• to inform and raise the expectations of higher education institutions, students, 
employers and other stakeholders about the processes and outcomes of higher 
education; 

• to contribute to a common frame of reference for the provision of higher 
education and the assurance of quality within the EHEA.” (ENQA 2005: 14) 

It is thus clear that the approach adopted by the authors of the ESG at the time was 
intended to be developmental. It was also hoped that newly emerging national and 
institutional quality assurance systems would be encouraged to take account of what 
was considered to be good practice in the field of quality assurance (QA). Although 


9 


at first some of the standards or guidelines were considered by some to be radical or 
controversial, six years on they are no longer perceived as such. 

Changing context 

Since 2005 considerable progress has been made in putting the Ministerial 
Communique commitments into practice at national level. When ministers officially 
launched the EHEA in 2010, QA processes had been implemented in all Bologna 
signatory countries, to various degrees and in various ways. (Westerheijden et al, 2010, 
30) 

Aside from the impact of the ESG which will be discussed later in this report, there 
have also been significant developments in terms of other action lines of the Bologna 
Process: development of national qualification frameworks, shift towards student 
centred-learning with emphasis on learning outcomes, increasing internationalisation 
and setting of ambitious goals for mobility to mention just a few. Furthermore, an 
increased demand for transparency and information both at institutional and system 
level in interaction with Bologna tools and frameworks, including QA, is demonstrating 
the impact of the process. 

In addition to the other developments in QA, in 2005 the E4 Group proposed to the 
Ministers to establish a register of external quality assurance agencies operating in 
Europe. In response the Ministers stated that “we welcome the principle of a European 
register of quality assurance agencies based on national review. We ask that the 
practicalities of implementation be further developed by ENQA in cooperation with 
EUA, EUR ASHE and ESIB with a report back to us through the Follow-up Group. “ 
(BPMC 2005) In 2006 the idea of a Register was raised in the European Parliament 
and Council recommendation on further cooperation in quality assurance in higher 
education (EP 2006). In 2007 the Ministers endorsed the proposal of the same E4 
Group to set up a European Register of Quality Assurance Agencies (EQAR), which 
was established by the stakeholder organisations as an association under Belgian law in 
2008. 

In 2004, ENQA decided to use the (then still draft) ESG as a decisive part of its 
membership criteria. In addition, EQAR - the first legal body established within the 
Bologna Process - uses the ESG as criteria for inclusion in the Register, thus examining 
the compliance of agencies applying for inclusion with parts two and three of the 
ESG. These two factors have contributed to the view that the ESG (again, particularly 
parts two and three) are a compliance tool or checklist. The missions and functions of 
EQAR and ENQA are set out below in order to clarify and explain the differences (and 
similarities) between the two organisations. 

ENQA and EQAR 

The missions and functions of ENQA and EQAR are similar in some respects and differ 
in others. Both organisations adhere to the principles of diversity reflected in the ESG, 
which state that primary responsibility for quality lies with HEIs and that the assurance 
of quality should be carried out transparently by agencies who maintain independence 
in the operation of their review processes and the published outcomes of those 
processes. However, while EQAR’s objective - through managing a list of QA agencies 
that substantially comply with the ESG - is to enhance trust and confidence by 
providing clear and reliable information on QA agencies in the EHEA and to promote 
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the mutual acceptance of QA decisions, ENQA aims to contribute to the maintenance 
and enhancement of the quality of European HE and to act as a major driving force 
for the development of QA across all the Bologna signatory countries. Thus EQAR 
decisions reflect a “snapshot” of agencies at a given time, whereas ENQA aims to work 
with its members (and potential members) to enhance their operation and contribute to 
the continuous development of external QA systems. 3 

EQAR, therefore, is managed by the E4 organisations as founding members, in 
co-operation with government members and other social partners. The decisions on 
inclusion are made by an independent Register Committee. ENQA, on the other hand, 
is a membership organisation of QA agencies and is a representative of and managed by 
its members. 

Since 2008 two organisations have separately examined and judged the compliance 
of QAAs with the ESG. Their interpretations of the ESG - which, as mentioned, 
originally were not designed for this purpose - may have on some occasions also 
differed. This might be due to their different missions (EQAR being primarily the 
guardian of a list of trustworthy agencies and with a focus solely on substantial 
compliance with the ESG and ENQA with its more developmental role). Both 
organisations use the threshold of “substantial” compliance rather than requiring full 
compliance with all of the standards. However, it should be noted that, in the first cycle 
of reviews of agencies, discrepancies between the decisions of the two bodies were 
minimal. 

Discussion on the revision of the ESG 

The 2005 report on the ESG stated that they should not be considered as unchangeable 
(ENQA 2005: 13). Thus, it was always clear to the authors that once some time for 
implementation had elapsed, the ESG would need to be analysed and reviewed. 

The European quality assurance landscape has changed considerably in recent 
years. Moreover, the creation of EQAR with its specific mission and use of the ESG 
as a compliance tool has brought additional pressure to bear in considering whether 
the ESG can serve the purpose for which they are being used. This has already been 
acknowledged by the E4 organisations in their “Report to the London conference of 
Ministers on a European Register of Quality Assurance Agencies” in 2007. The report 
stated that: 

“It is probable that the ESG will need to be revised from time to time. They should 
not, however, be changed during the first two years of the Register’s existence, 
in order that there should not be uncertainty about the status of the criteria for 
inclusion. In normal circumstances, revisions might be expected every five years. 
Care should be taken to make sure that the European Standards and Guidelines 
do not become a simple checklist for compliance purposes and that any revisions 
reflect the needs of higher education more broadly. E4 proposes that responsibility 
for commissioning revisions to the ESG should rest with the BFUG or any 
successor body; the BFUG should request E4 to recommend revisions when E4 
considers it appropriate to do so. Any proposals for revision of the ESG should 
be undertaken in consultation with ENQA, as they are the basis of the ENQA 
membership criteria.” (ENQA 2007: 14) 


3 See the missions of the two organisations on their respective websites www.eqar.eu and www.enqa.eu 
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In 2009 the European Commission published a “Report on progress in quality 
assurance in higher education” which included a recommendation to revisit the ESG. 
(EC 2009: 5) Since the report was published, the possibility of revising the ESG has 
been addressed systematically in various meetings, events and forums attended by all 
stakeholders (representatives of HEIs, QA agencies, students, governments, employers 
etc.) 4 . However, to date there has been no consensus on whether or not this revision is 
necessary, nor in which direction it should go. 

One clear deficiency identified through the discussions was that there had been 
no overarching study on the implementation of the ESG in all its parts, and that no 
thorough analysis of the impact of the ESG had yet been carried out. Discussions 
were often focused on a single aspect or part of the ESG, without a thorough, all- 
encompassing discussion of the essence of the document, its purpose and how various 
parts interact with, and balance each other. 

With this in mind, the E4 Group decided to launch the “Mapping the 
Implementation and Application of the ESG (MAP-ESG)” project in order to gather 
information on how the ESG have been implemented and applied in the 47 Bologna 
signatory countries in HEIs and in QAAs, and to launch a thorough discussion on the 
ESG in their entirety. 

This report presents the results of the activities carried out in the context of the 
MAP-ESG project. The following chapter describes the project in further detail: its 
purpose, activities and actors. The third chapter discusses the key findings of the 
project and analyses the results, while the conclusions include the recommendations of 
the project consortium to the Bologna Follow-up Group and the Ministerial meeting in 
Bucharest in April 2012. The annexes present the key reflections from the four parallel 
consultations that took place in the framework of this project during spring 2011. 


4 See for example the European Quality Assurance Forum 2010, http://www.eua.be/EQAF-Lyon.aspx 
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2. Introduction to the MAP-ESG project 


Mandate 

At the Bergen meeting (2005) the Ministers adopted the European Standards and 
Guidelines for QA in the EHEA, as proposed by ENQA, in cooperation with ESU, EUA 
and EURASHE. 

In the London 2007 communique (2.12) the Ministers recognised that the 
“Standards and Guidelines for Quality Assurance in the EHEA have been a powerful 
driver of change in relation to quality assurance”. It was further said that “all countries 
have started to implement them and some have made substantial progress” (Bologna 
Process Ministerial Communique, London 2007). 

The E4 organisations, as the authors of the ESG and the founding members of 
EQAR, felt responsible for continuing the process of enhancing common European 
approaches to QA. At the beginning of 2010, after five years of use, they agreed that a 
mapping exercise of the implementation and application of the ESG was necessary in 
order to ensure that the ESG still meet the current challenges of higher education. 

The E4 therefore decided to apply for funding from the European Commission to 
carry out a joint project, which would investigate the current state of implementation, 
and on the basis of the results of their consultations, make a recommendation as to the 
necessity and scope of a revision of the ESG. 

Role of the partners of the project consortium 

The four partners (which represent the main stakeholders in QA) consulted their 
respective membership, and drew conclusions and recommendations from those 
consultations, which then fed into a joint report to the Ministers. A Steering Group, 
with representatives of the four organisations, and for which ENQA volunteered to 
provide the Secretary, monitored the different stages in the project and the production 
of the final joint report, which is complemented by the individual reports of the four 
organisations as annexes. 

In order to ensure wider involvement of other stakeholders, the project was assisted 
by an Advisory Board, consisting of EQAR, Business Europe, Education International 
and government representatives identified by the BFUG. 

Methodology 

Each member of the E4 group carried out a consultation with its members, the 
methodology of which is described in this chapter of the report. In addition, there is a 
brief note as well on EQAR’s input. 

ENQA CONSULTATION 

ENQA’s consultation was carried out at two levels: 

• A questionnaire was sent to all ENQA members and a slightly abridged version 
of that questionnaire was sent to the EQAR Register Committee and the Review 
Committee of the ENQA Board. The questionnaire was designed to contain 
both open and closed questions in order to allow respondents to be discursive in 
their responses. The responses to these questions yielded the findings that were 
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described in ENQA’s interim report, which was drafted by a small working group 
consisting of the President and one of the Vice Presidents of ENQA and a former 
ENQA Board member. 

• A meeting of a group of ENQA experts, who analysed, tested and challenged the 
findings contained in the draft report. 

ENQA was also able to draw on a report it had produced last year on the use of the ESG 
in agency reviews as background material. 

ESU CONSULTATION 

The aim of the ESU consultations on the ESG was to bring together the multitude of 
student experiences in quality assurance processes with a particular attention on the 
impact and use of the ESG in different contexts - national, institutional and within 
external reviews. 

Through the consultation process ESU wanted to evaluate the ESG in terms of their 
understandability, clarity and completeness as well as to map their implementation and 
impact on the institutional, national and quality assurance agencies level. 

The research team was responsible for conducting the ESU consultation, the 
development of the consultation methodology and instruments, and for analysing and 
interpreting the data and writing the ESU consultation report. 

The ESU consultation process aimed at building on the experiences of national and 
local student organisations, student reviewers who have been involved in the external 
quality assurance of quality assurance agencies and individual student representatives 
who have gained experience in quality assurance on the institutional, national or 
European level and who qualify as members of the ESU quality assurance student 
experts’ pool. Individual members of the student experts’ pool took part in a MAP- 
ESG consultation workshop in Mainz, Germany; then between May and July 2011 45 
national unions of students from 38 European countries (all the members of ESU) were 
consulted using a structured questionnaire set up online. In total, 29 national students’ 
unions from 27 countries answered the questionnaire, which is a response rate of 
64,5%. The responses were handed in anonymously. 

With the aim of getting a perspective of higher education institutional quality 
assurance all over the EHEA, the research team also decided to consult a sample 
of local student representatives. In order to increase accessibility, the structured 
questionnaire was available in six languages (English, French, German, Serbian, 
Slovenian, and Spanish). 

To receive firsthand experience of the external reviews of quality assurance 
agencies, 28 students who had been members of panels reviewing quality assurance 
agencies were consulted through a structured questionnaire in June and July 2011. 

The questionnaire addressed the individual experience of the students in the external 
review team, relevance and adequacy of the ESG as a whole and individual standards 
and guidelines in the reviews of QA agencies; it also inquired about the views of the 
students on the necessity and the way in which the ESG could be revised and/or 
improved. 

The information obtained through questionnaires, interviews and focus groups was 
complemented with text-analysis of the ESG and a review of relevant documents on QA 
and the ESG. 
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EUA CONSULTATION 

EUA’s consultation among its members consisted of two parts: desk research and 
workshops. 

Desk research looked into existing information sources, such as the results of 
previous studies conducted by EUA (Trends series, Examining Quality Culture survey). 
As part of this desk research, EUA also disseminated another concise survey to each 
of its collective members (the national rectors’ conferences from 34 countries). This 
survey focused on the changes in the national or regional quality assurance frameworks 
and on the collective view on how the current state of play - with regard to external 
quality assurance in particular - reflects the ESG. 

In addition, a series of workshops was organised offering the participating 
universities opportunities to benchmark their quality assurance processes in teaching 
and learning against the ESG, share their experiences of good practice as well as 
obstacles encountered in implementing these processes, and provide feedback on the 
ESG. 

Further, to the specific themes of each workshop, the programme was designed 
to encourage the participants also to discuss the role of the ESG on a wider scale 
and to reflect on the other sections of the document. This allowed the organisers to 
triangulate the data collected. 

Following a call for interest, 49 delegates from 23 European countries participated in 
the workshops, thus contributing to a thorough discussion on how the ESG have been 
implemented and applied in institutional contexts. 

EURASHE CONSULTATION 

EURASHE’s consultation was not limited to its members’ input only, although the large 
majority of its respondents was from or had an affinity with the sector of professionally 
oriented programmes and institutions. 

The consultation process consisted of four main parts: an on-line survey via 
EURASHE’s website (with a set of about 30 questions) addressing all stakeholders, who 
had to identify themselves as respondents from institutions, students, stakeholders, etc. 
when answering the questionnaire. 

Simultaneously a direct mailing was directed to ‘institutional members’ via the 
Council of EURASHE, in which specifically heads of institutions were addressed, 
who were asked to identify also other respondents in their own environment, and 
an ‘external stakeholder’ (not directly linked to their institution), such as employers, 
alumni, student counselors, etc. 

It was not possible, however, to maintain such consistency in the later phases of the 
consultation, which therefore limited the full institutional approach to a few countries. 
Still the picture resulting from these cases confirmed the overall results of the on-line 
survey. When concluded over 230 fully completed and ‘valid’ responses from 26 
countries in the EHEA had been received. 

Next came a consultation in the form of workshops at a members’ event, the 
annual EURASHE conference on 1 April 2011. 

It was then decided to add another consultation method, namely structured 
telephone and face-to-face interviews (called ‘in-depth interviews’ as the target group 
was this time, a sample of those that had responded on-line).This selection of identified 
‘interested persons’ in a number of countries were given targeted and ‘open’ questions 
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which this time geared more to the implementation and the review’ perspective of 
the ESG. For this part of the consultation an attempt was made to cover different 
geographical and cultural areas in the EHEA, based on presumptions from the on-line 
consultation that the geographical and cultural spread was an important element in 
assessing the degree of both knowledge and implementation of the ESG. 

The different approaches, using varying sources of consultation, resulted in a 
well-documented and extremely diverse set of findings, which helped provide a 
comprehensive picture of the ESG implementation for the entire professional sector of 
the EHEA, covered by EURASHE. 

The results of the consultations and the ensuing interim reports were used by the 
Steering Group, through an iterative process, to produce the findings and agree the 
conclusions set out in this report. 

EQAR CONTRIBUTION 

EQAR contributed to the project as a member of the Advisory Board. It provided 
written input on ‘strengths and challenges’ when using the ESG in evaluating 
applications from 35 QA agencies since it opened for applications in August 2008. 
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3. Major findings of the project 

This chapter of the report synthesises the main findings of the consultation processes 
carried out by the MAP-ESG project partners (ENQA, ESU, EUA and EURASHE). 
Information has been organised under three headings (Purpose and scope of the ESG, 
Clarity and usability of the ESG and Implementation and impact of the ESG); the key 
points under each heading are set out in a way that allows the reader to see where 
there is agreement amongst the E4 and also where different views were apparent in the 
results of the consultations, showing the different perspectives and expectations of the 
four stakeholders. 

In general, the responses to the consultations show that the ESG are perceived to be 
a useful and useable tool which has been widely implemented across the EHEA and has 
had a significant impact on the development of QA both as a European-wide concept 
and in individual national contexts. 

Purpose and scope of the ESG 

PURPOSE 

The findings of all four consultations demonstrate that the ESG, in addition to their 
original and primary purpose of providing generic principles rather than specific 
procedures, are perceived to fulfil the purposes for which they were originally intended, 
but also a range of others. Their general relevance as a primary reference point for 
internal and external quality assurance is undisputed and there was agreement that the 
original purposes and objectives (see chapter one for detail) are appropriate, although 
there should be consideration of the change in context since the time of writing (see 
below). The MAP-ESG project began five years after the production of the ESG - there 
is a need, therefore, to review the purposes and objectives the ESG are serving. 

The purposes of the ESG as currently stated were clear for most respondents; there 
was a preference amongst many respondents for them to be regarded as a reference 
point and not a checklist or compliance tool. They are widely regarded as being a 
successful reference framework for the implementation of quality systems at national 
and institutional level. This was particularly stressed by respondents to the EUA and 
ESU consultations, although EUA noted that they are not necessarily a direct source 
of reference for many institutions, which need to assure that their internal processes 
comply with the requirements of the external quality assurance system with which they 
work. However, since most external QA systems are in line with the ESG, the impact 
on institutions is indirect. 

The ESG as a means of providing a common framework for QA in the EHEA was 
agreed to be a sound purpose, although ESU respondents doubted whether ESG as they 
currently stand really help to provide greater transparency or easier understanding of 
external QA or if they improve the education available to students indirectly via the 
QA systems for which they provide a framework. The ESU consultation reported the 
view that that the standards in part one of the ESG appear to be more clearly defined 
in the national quality assurance systems for internal QA than those in part two are 
for systems of external QA. However, this does not necessarily mean that they are 
implemented in individual HEIs, suggesting that there might be a mismatch between 
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national legislation and its implementation. Students also put forward the view that 
existing national standards for external QA differ in some important respects from the 
standards set out in part two of the ESG. 

There was a general agreement that it is crucial to maintain the ESG as generic 
principles as these can be adapted to various political, national and cultural contexts 
and respect system level and institutional diversity - one of the strengths of the 
EHEA. This was of particular importance to the participants in the EUA consultation. 
However, it was pointed out by respondents to the ESU consultation that the ESG also 
need to be concrete enough, for example in standards dealing with the publishing of 
reports and follow-up procedures to ensure comparability of external QA systems thus 
indirectly facilitating comparability of outcomes of different HE systems and, therefore, 
ease of mobility. ENQA, EUA and EURASHE respondents, whilst not disagreeing with 
this aim, emphasised that such comparability would need to be at a broad level in order 
to ensure that diversity is still respected. 

Despite the general agreement around the purposes of the ESG, one significant 
point, particularly to do with part three of the ESG (standards and guidelines for 
the review of QAAs), was raised by the respondents to the ENQA consultation and 
recognised by respondents to the other consultations as well as the EQAR contribution. 
The issue is that of the tension between the original intentions and purposes of the 
ESG and what they are currently used for, that is, a compliance tool. Whereas EQAR 
uses the ESG solely as a tool for inspection of compliance as a basis for the decision on 
whether or not an agency is listed on the Register, ENQA uses them as a developmental 
tool and as a basis for recommendations as to how agencies can further develop, as 
well as a means for making decisions on membership within ENQA. This tension came 
across very strongly in the ENQA responses but did not detract from their view that it 
was preferable that the purpose of the ESG is as a reference point and not a compliance 
tool. Paradoxically this tension could also be viewed as a strength of the ESG since they 
seem to be able to accommodate both purposes although perhaps not perfectly. The 
same degree of tension was not perceived in relation to part one of the ESG. 

In common with the E4 consultations, the EQAR contribution to the project 
highlighted the need to ensure that the ESG are as clear and unambiguous as possible 
in order to facilitate the process of making judgements. ESU also raised the need for 
a discussion on how to bridge the discrepancy between the ESG as providing criteria 
for reviews of HEIs (through part two) but not being used as a compliance tool. The 
consultation therefore suggested the development of a guide on how to use the ESG 
for external reviews or an explanatory report similar to that of the Lisbon Recognition 
Convention in order to improve usability in external reviews. 

SCOPE 

At present, the scope of the ESG encompasses principles for managing the assurance 
of quality in HE, both internally and externally and for QAAs. Only the respondents 
to the ESU consultation saw the need to broaden and clarify the scope of the ESG. 
Other members of the E4 group cited several reasons for maintaining the current 
scope. These included: the belief that further areas requiring consideration would 
lead to a significant increase in bureaucracy, the fear that it would lead to a check list 
approach and a possible negative impact on institutional responsibility for the quality 
of education that they offer due to the possibility of an increasingly intrusive role for 
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external QA. For EUA, the final point - that institutions are responsible for quality - is 
considered a core belief that should be made more explicit in the ESG. 

Widening the scope to include, for example, research, institutional management 
or service to society, could be seen as difficult in terms of coming to a consensus 
on principles for QA in these fields. The EUA consultation concluded that it would 
probably not be feasible given the present responsibilities and modalities in place in 
national systems and questioned whether this kind of approach would gain acceptance 
from all parties concerned. Nevertheless it should be possible to refer to them in any 
revised introduction to the ESG. There was, however, some agreement that the strong 
link between research and teaching might be made more explicit within the ESG. 
Opinions differed as to whether it would be possible to do this by referring, in the ESG, 
to existing reference points in this area. However, a broad consensus was reached that 
the ESG should make the relationship with other Bologna tools such as qualification 
frameworks and the European Credit Transfer System (ECTS) more explicit. 

ESU respondents thought that there was no original mandate regarding the scope of 
the ESG and that the current edition does not clarify how the scope was determined. 
They believe that the scope reflects that of the Bologna Process in 2004. While agreeing 
that the ESG should not become a policy evaluation tool (which is also a key message 
from the other 3 stakeholder respondents) student respondents found that the ESG do 
not currently reflect aspects of quality in HE that would be within the remit of quality 
assurance. In relation to this ESU suggests widening the scope in two ways: 

i. By making the wider Bologna principles (e.g. student-centred learning, the social 
dimension of education and lifelong learning) more explicit in the ESG and 

ii. By recognising more explicitly the specific commitments of the Bologna Process 
(e.g. learning outcomes). 

Both of these would involve either amending the current standards and guidelines or 
developing new ones. Some of these points are currently referred to in the introductory 
parts of the current edition but ESU respondents stated that this does not give them 
sufficient emphasis in terms of their actual use in national and institutional QA 
systems. 

The EUA consultation also recognised that any revised ESG should reflect trends 
that have strengthened since 2005, namely the importance of transparency, student- 
centred learning, widening access and promotion of the European dimension of QA. 

But it concluded that this could be done by reformulating the existing standards and 
maintaining the ESG as generic principles. 

EURASHE’s consultation revealed that the relationship between the ESG and the 
Bologna Process and its tools as a whole was not always clear to stakeholders, although 
it was generally acknowledged that there is a link. 

Both the EUA consultation and EQAR’s input to the project noted that the ESG do 
not explicitly define “QA agencies” or “external QA processes”. (This was not mentioned 
as a problem by agencies in response to the ENQA consultation). They do, however, 
acknowledge the diversity of QA purposes and approaches. This acknowledgement was 
considered important by EUA respondents who saw it as being crucial to the further 
development of the European dimension in quality assurance and to EQAR. The EQAR 
input, however, questioned the limits of this diversity and called for a policy discussion 
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on the topic, which would impact both the required characteristics of the ESG as well 
as the role of EQAR. 

The EUA consultancy and the EQAR input highlighted that some of the standards, 
namely 2.6 (follow-up procedures) and 2.7 (periodic reviews), can only be ultimately 
“enforced” by laws or by an agency with its own formal authority. However, in a 
system where institutions can turn to several agencies, which is one of the reasons 
underpinning the establishment of EQAR, or where an agency is not a formal authority 
but an agency commissioned to carry out an external QA process on voluntary or 
temporary basis, the agency itself cannot effectively ensure periodicity of reviews or 
enforce follow-up procedures. 

Clarity and usability of the ESG 

Respondents to all four consultations found the ESG to be well regarded and reasonably 
easy to use. EURASHE respondents felt that the usefulness/usability of the ESG can 
depend on the national context and its history of QA to date. In other words, if a 
country has a long history of QA in HE and has developed and refined criteria and 
external methodologies by which HEIs and/or their programmes are evaluated, it 
is likely to have a different perspective on the usefulness and usability of the ESG 
compared with a country with a very recent history of work in this area. Moreover, 

EUA respondents found the ESG part one on internal QA easy to use precisely because 
of their nature as a reference point, but acknowledged that, when considering the work 
of QA agencies, a different tension probably exists as the ESG can be considered as a 
checklist. 

ENQA respondents felt that the ESG are used reasonably consistently in the external 
reviews of agencies (ENQA also has evidence for this also from its project “An analysis 
of agency reviews” 2010). 

However, respondents to all consultations, including the EQAR contribution, also 
described various difficulties in relation to the clarity of the ESG. There was agreement 
that the document needs to be updated and that there are issues around terminology 
and interpretation of language and concepts. 

ESU commented that many of its respondents, who had been reviewers on 
panels for the review of agencies, whilst finding the ESG easy to use from an 
individual perspective, also found that there was often no common understanding 
or interpretation of the ESG amongst panel members; thus there is a danger that 
they are not interpreted and applied consistently. Agencies also feel that there is 
some inconsistency in interpretation of the ESG by various panel members. The 
possibilities of differences in interpretation are not only due to differing views of 
various stakeholders but also to differences in institutional and national contexts. This 
point becomes all the more important if the ESG are to have multiple purposes. It 
should be noted that, in the view of the Steering Group, consistency of interpretation 
is a matter for the E4 group to consider, perhaps by developing a joint work plan to 
facilitate awareness and a common understanding of the ESG. However, it is clear 
from responses to the consultations that any revisions to the ESG should ensure that 
all possible ambiguities and unclear text are dealt with in a way that makes the ESG as 
clear as possible to all stakeholders and users of the document. 

Responses to the ESU consultation continued the theme of inconsistency by 
providing evidence that the impact of the ESG on student involvement or participation 
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in QA at various levels is inconsistent (although this is not necessarily the fault of the 
ESG themselves). Even terminology like “student involvement”, “student participation” 
and even “student feedback” can lead to different interpretations of the extent to which 
students should be involved in both external and internal QA processes. They believe 
that this further strengthens the argument that revisions to the ESG are needed in 
order to ensure more consistency of application, implementation and impact, despite 
national contextual differences. 

ENQA and ESU respondents suggested a glossary as one way of helping to overcome 
inconsistency of interpretation. The terms “standards” and “guidelines” are two 
examples that are cited as needing further clarification. Clearer demarcation is felt to 
be needed since currently some of the guidelines are interpreted as standards, some 
standards do not have guidelines and some guidelines are much more detailed than the 
accompanying standard or than the guidelines of other standards. Some standards are 
also written more precisely than others. 

ENQA and ESU respondents also asked whether their respondents perceived some 
standards are more central than others and whether they are all given the same weight 
in reviews of both HEIs and agencies. It was felt that it would be useful to have a 
discussion on core standards for which compliance overall is necessary or whether all 
the standards are of equal importance. This also relates to the notion of substantial 
compliance used by ENQA and EQAR. Although this issue may not seem to directly 
impact on part one of the ESG, nevertheless, if there is an implication that there is 
a compliance approach to part two, this would, necessarily impact on part one and 
how it is perceived and used by HEIs. The EUA response to this particular question 
noted that the importance of individual standards might vary from one institutional 
or national (in the case of part two of the ESG) context to another and that national 
policy discussions might also lead to changes in priorities. ESU respondents, while 
also recognising this, felt that this flexibility includes a potential danger of shifting 
expectations regarding QA systems; students see the ESG as a mechanism to prevent 
politically motivated changes to national systems and thus as providing constancy in 
those systems. 

Implementation and impact of the ESG 

Responses to questions on the implementation and impact of the ESG naturally 
returned to points that have been made above since it is at the practical level of 
implementation that the impact of some of the views expressed in earlier sections 
becomes practice. There is, therefore, some repetition in this section but with a focus 
on the ESG as a practical document rather than a theoretical framework. 

IMPACT 

The ESG have clearly impacted on QA in the EHEA in various ways and at several 
levels, both directly and indirectly. There is evidence to suggest that, for example, 
in some national contexts, legislation has been drafted to ensure that the ESG are 
enshrined in the procedures for QA in higher education, in others new QAAs have 
been created or existing QAAs have ensured the implementation of part one of the ESG 
through their external QA processes. Students are increasingly involved in external 
and internal QA processes, although the ESU consultation shows that there is a great 
variety in the degree of student involvement in internal QA; on average, students 
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are less involved in internal QA than in external QA processes. Finally, but perhaps 
most importantly in terms of the ESG impacting on HEIs’ primary responsibility for 
QA, HEIs have developed QA systems that take into account the ESG. As the EUA 
consultation demonstrated, great progress has been made in terms of developing 
internal QA, in particular after 2005. 

Thus many changes and developments in relation to QA in institutions and in 
national systems have been due to the introduction of the ESG and even where they 
are not directly linked to changes, they can be seen as underpinning developments 
or as providing a reference point against which well-established systems can compare 
themselves. Some of these changes are discussed in further detail in the four 
consultation reports which are appended to this report. 

Individual national contexts are important when discussing the implementation 
and impact of the ESG; respondents to all consultations agreed that it is crucial that 
the ESG can be implemented in diverse contexts. The impact that they have may also 
depend on national contexts and on each country’s history of QA - both ESU and 
EURASHE respondents agreed that greater or lesser (perceived) impact might be to do 
with either national legislation or whether the QA system is new or already embedded. 

From the ENQA perspective, the fact that its respondents mainly found the ESG 
easy to implement is a measure of their success - some agencies also apply them in an 
non-European context with similar ease of application and use. However, if there were 
problems in implementation, this was generally due to national legislation (usually 
in the areas of independence and resourcing). For ENQA this strengthened the view 
that anything that shifted the purpose and scope of the ESG to a long and exhaustive 
checklist would make the tension between complying with the ESG and national 
legislation more difficult. There was also strong agreement amongst ENQA respondents 
that the standards should not be broken down into more detailed requirements for 
operational purposes. 

The impact of the ESG in reviews of agencies is two-fold; they have had a clear 
impact on the work of agencies as described above; the reviews have acted as a tool for 
ensuring their impact on agencies. In terms of reviewing agencies against part three of 
the ESG, the reviews are believed to have many benefits, for example, the opportunity 
to reflect on the work of the agency through the production of a self evaluation 
document and to benchmark it against agreed principles of good practice. In general, 
agencies did not consider compliance with the ESG as a particular benefit of the review, 
preferring to cite the more developmental aspects of the process in terms of self- 
reflection and the receipt of a report highlighting features of good practice and making 
recommendations. 

The EQAR input to the project concluded that the ESG were applicable to and 
impacted on direct external QA activities such as external criteria and processes used 
by agencies. However, it found that that they were not so applicable to meta-level 
activities and suggested that there should be clarification as to which activities are 
addressed by the ESG. 

IMPLEMENTATION 

EUA respondents noted that the ESG are widely accepted and adopted but that the 
challenge lies in the implementation - in other words, how to design and carry out 
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processes that are in line with these principles and are effective and supportive to the 
development of institutions. 

It was pointed out by ESU respondents that the link between awareness of the 
ESG and their implementation is important. Perception of levels of implementation 
can vary amongst the different stakeholders, e.g. students might be more aware of 
implementation at institutional level or, if they are officers at national level, of part 
two of the ESG. The student respondents demonstrated a strong perception that 
implementation is due to HEIs rather than governments and that, whilst in one HEI the 
ESG might be well-implemented, they could be less so at the neighbouring HEI because 
of a lack of national encouragement. This point was also recognised by EUR ASHE 
respondents. 

Any revision of the ESG needs to bear in mind that levels of implementation vary 
greatly and to ensure that national and institutional level stakeholders are aware of the 
purposes of the ESG. This should take account of the fact that currently, awareness 
amongst faculty, apart from those directly involved in QA work, is very low. 

Clarity of terminology arose again under implementation and impact. However, 
despite some confusion around, for example, “substantial” and “partial” compliance in 
relation to reviews of agencies, agencies resisted the idea of a threshold for compliance 
with each standard, believing it would lead to an unmanageable level of bureaucracy. 

In summary, the fact that the ESG have had a considerable impact on the 
development of national and institutional QA systems does not necessarily mean that 
there is a high level consistency in their application either at institutional, system or 
national level. 
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4 . Conclusions 


As is clearly evidenced by the findings of the MAP-ESG project, the ESG have proved 
to be a major achievement of the Bologna Process - they are well-regarded by all 
stakeholders, have proved to be applicable across diverse contexts and have impacted 
on the development of QA processes at institutional and national level and on the work 
and review of QAAs. They have facilitated a shared understanding of QA amongst 
the relevant stakeholders and actors in HE, and QA, whether internal or external, is 
conducted across the EHEA according to the framework of principles they provide. 

They constitute a crucial means for achieving the goals of the Bologna Process. 

The conclusions of this project, therefore, focus on the improvement of the ESG as 
they are currently formulated rather than to recommend a wholesale revision of the 
content of the principles enshrined in the document. There is, therefore, no intention to 
suggest changing direction towards either broader standards or more of a checklist. 

Purpose and scope of the ESG 

It was generally concluded that it is essential to maintain the concept of the generic 
principle in order to ensure the continuing relevance of the ESG to all relevant 
stakeholders in the EHEA and to maintain their authority as the common reference 
point for QA in the EHEA. However, it is more difficult to come to any conclusion 
around the area of tension regarding the ESG as a developmental reference point and 
also as a compliance tool. 

In terms of the scope of the document, there was agreement that the current scope 
is generally appropriate but there is encouragement to reflect on the extent to which 
a revised ESG document should link to specific Bologna commitments and reflect 
overarching principles agreed among the Bologna signatories. 

Clarity and usability 

It was apparent that the ESG are widely used across different contexts and at all levels. 
However, it was also clear that the document could be improved by some further work 
to increase clarity of terminology and to ensure the removal of ambiguity both in terms 
of the language used and also with regard to the standards and guidelines themselves to 
ensure that they are as clear as possible. 

Impact and implementation 

It can generally be concluded that there is clear evidence that the ESG have been widely 
implemented and have impacted significantly on QA in the EHEA. Impact can be seen 
in processes for both internal and external QA and also at national and system level. 
However, further work could be done to raise awareness and ownership of the ESG 
even further, particularly amongst faculty staff directly involved in the student learning 
and teaching process. 
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5 . Recommendation 


Throughout the course of its work, the Steering Group was very aware that, in the 
case of any revision to the ESG, it is important to remember the impact of changes on 
those stakeholders who use the document on a regular basis as well as on a number of 
countries, HEIs and agencies that have only recently begun to work with the ESG and 
to implement them. With that point in mind, the Steering Group makes the following 
recommendation to ministers: 

Based on the findings of the project, “Mapping the implementation and application 
of the ESG” and the conclusions derived from the findings, ENQA , ESU, EUA and 
EURASHE recommend that ministers of the EHEA countries mandate the E4 organisations 
to carry out , in consultation with all relevant stakeholders , notably Education 
International Business Europe , the Bologna Follow Up Group (BFUG) and EQAR , a careful 
revision of the ESG in order to improve their clarity , applicability and usefulness. This work 
would be carried out in the understanding that the current principles would be maintained. 
The report , with the revised document , would be referred to the BFUG. 

The detailed work carried out in the MAP-ESG project brought into focus different 
views, perspectives, contexts and expectations with regard to the ESG and this is why 
the Steering Group recommends that some further work to revise the ESG should be 
carried out. Experience has shown that the success of the ESG lies in the fact that they 
were drafted in such a way as to reach consensus; any future edition will require the 
same level of consensus and the Steering Group wishes to ensure the input of as wide a 
range of relevant stakeholders as possible. E4 will suggest a timeline that allows for full 
consideration of all input, followed by detailed redrafting. The Steering Group does not 
underestimate the task of bringing together those different views and expectations into 
a revised and improved version of the ESG; the E4 organisations are willing to take up 
this challenge and are confident that, in co-operation with all stakeholders, it can be 
achieved. 
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Annex 1: Report on the ENQA 
consultation 


Executive summary 

The E4 group (EUA, EURASHE, ESU and ENQA) is currently engaged in the MAP-ESG 
project which will report to Ministers at the Bologna meeting in Bucharest in March 
2012 as to whether or not the Standards and Guidelines for Quality Assurance in the 
European Higher Education Area (ESG) should be revised. 

Each member of the E4 group has carried out a consultation with its members; this 
is the report of the findings of the ENQA consultation and an analysis of those findings. 
ENQA’s consultation was carried out at two levels: 

• A questionnaire was sent to all ENQA members and a slightly abridged version of 
that questionnaire was sent to the Register Committee of EQAR and the Review 
Committee of the ENQA Board. The responses to these questions yielded the 
findings that are described in the main body of this report. 

• A meeting of a group of ENQA experts who analysed the findings contained in the 
draft report and developed proposals (set out at the end of this report) based on 
that analysis. 

The main headlines of the findings can be summarised as follows: 

• It is clear from the responses to the consultation that the ESG are well-regarded, 
used reasonably consistently and do not present any major problems in terms of 
their usability or applicability; 

• Whilst there were comments around the consistency of knowledge, understanding 
and application of the ESG, the majority opinion was that it would not be timely 
to revise them substantially in the near future; 

• The majority of agencies are aware of the tension between the original intention 
and purpose of the ESG as a non-compliance document and their actual use as a 
compliance measure for ENQA membership purposes and/or inclusion in EQAR; 
however, they consider the ESG as a reference point and would like to be able to 
use them as such; 

• There could usefully be some developmental work to update the document; this 
might include clarification of terminology (including a glossary) and some further 
clarification of the terms ‘standards’ and ‘guidelines’ (i.e. their status and role and 
the difference between them); 

• The majority of agencies feel that the scope of the ESG is appropriate and would 
discourage any attempt to make them more detailed or to turn them into a check- 
list. 

The majority of agencies believe that additional beneficial work could be done to 
further mutual understanding of the ESG and consistency of application across 
stakeholders. 

ENQA’s response to the headlines in terms of its recommendations and the proposals 
of the expert group, based on its analysis of the findings can be found at the end of this 
report. 
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Prelace 

The E4 group (EUA, EURASHE, ESU and ENQA) is currently engaged in the MAP-ESG 
project which will report to Ministers at the Bologna meeting in Bucharest in March 
2012 as to whether or not the Standards and Guidelines for Quality Assurance in the 
European Higher Education Area (ESG) should be revised. 

The E4 group consulted individually with their respective members; this report is 
the outcome of the ENQA consultation process. 

Methodology 

ENQA was able to draw on several levels of expertise in gathering information from its 
members about the use and applicability of the ESG: 

1. It gathered information from its members via a questionnaire which was made 
available through SurveyMonkey. The questionnaire, designed by a working 
group consisting of the President, one of the Vice Presidents and a former ENQA 
Board member, was designed to find out members’ views on the purpose, scope 
and applicability of the ESG and contained both open and closed questions in 
order to ensure that members could be discursive in their answers. It addressed 
all three parts of the ESG, as had been agreed by the E4 Steering Committee for 
the project. A similar questionnaire was sent to the EQAR Register Committee 
and to the ENQA Board Review Committee. Only four replies were received from 
these sources and the results were similar to those obtained through the main 
questionnaire. Therefore these results are not included in the findings set out in 
the main body of this report. 

2. The results of the consultation were analysed by the working group and a draft 
report on the findings was produced. The findings were then tested, analysed and 
challenged by a group of experts in order to help finalise the report. 

3. ENQA was also able to draw on a report it had produced last year on the use of 
the ESG in agency reviews as background material. 

Headlines from the findings 

54 replies were received from ENQA full and candidate members, affiliates and 
associates. This is a high response rate of 63% and the working group is grateful to 
ENQA colleagues for taking the time to provide such detailed and useful responses to 
the questionnaire. 

The main headlines could be summarised as follows: 

• It is clear from the responses to the consultation that the ESG are well-regarded, 
used reasonably consistently and do not present any major problems in terms of 
their usability or applicability; 

• Whilst there were comments around the consistency of knowledge, understanding 
and application of the ESG, the majority opinion was that it would not be timely 
to revise them substantially in the near future; 

• The majority of agencies are aware of the tension between the original intention 
and purpose of the ESG as a non-compliance document and their actual use as 

a compliance measure for ENQA membership purposes and/or inclusion in the 
EQAR; however, they consider the ESG as a reference point and would like to be 
able to use them as such; 
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• There could usefully be some developmental work to update the document; this 
might include clarification of terminology (including a glossary) and some further 
clarification of the terms ‘standards’ and ‘guidelines’ (i.e. their status and role and 
the difference between them); 

• The majority of agencies feel that the scope of the ESG is appropriate and would 
discourage any attempt to make them more detailed or to turn them into a check- 
list; 

• The majority of agencies believe that additional beneficial work could be done to 
further mutual understanding of the ESG and consistency of application across 
stakeholders. 

Findings 

THE PURPOSE AND SCOPE OF THE ESG 

The responses to this section of the questionnaire went straight to the heart of ENQA 
members’ views on the ESG and it is clear that some of the issues raised in this section 
become themes that run through the outcomes of the rest of the consultation. 

Members’ views on the role and purpose of the ESG immediately raised the 
relationship between the ESG, ENQA and the European Quality Assurance Register 
(EQAR). It is clear that the majority of respondents (wish to) use the ESG in the 
spirit of their original purpose as set out in the "Context, aims and principles" of that 
document (“In both the standards and the guidelines , the report prefers the generic 
principle to the specific requirement”). However, responses indicate that, whatever the 
original intention of the ESG, the presence of EQAR and the use of the ESG as ENQA 
membership criteria means that many agencies feel they must regard the ESG as 
requirements to be fulfilled (“We use them as requirements since they turned out to play 
such a role. In principle we would prefer using them as a reference point”) 

This relationship with EQAR is critical - despite the presence of the Register, almost 
two thirds of respondents prefer to see the ESG as a set of reference points rather than 
rules and the contradiction between the original intention of the ESG and how they are 
currently used is not lost on agencies. (Certainly responses to questions about the scope 
of and possible revision to the document (see below for detail) suggest that there is 
currently no need to carry out any major revision to the ESG and certainly not to their 
purpose or scope). 

With regard to the scope of the ESG, a minority of respondents (18.5%) felt that the 
ESG could be broadened to include some of the other Bologna action lines. However, 
the majority were in agreement that the current scope was satisfactory as it was; one 
respondent even suggested that it is already too extensive. 

It is clear from the comments about the purpose and scope of the ESG that agencies 
feel that these are appropriate as they currently stand and that there is further 
developmental work to be done in terms of knowledge about and use of the ESG in 
their existing format (See recommendations at the end of the report). 

APPLICABILITY OF THE ESG 

There is further reflection of the agencies’ views on the tension between reference point 
and requirement in their responses to the applicability of the ESG. Several responses 
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suggested that the “principle-based” approach is not clear and asked for further 
clarification of the role of principles. 

In terms of linguistic comprehension and applicability of the ESG, not many 
problems were presented in the responses. The most problematic term seems to be 
‘substantial compliance’ with 16 respondents (34%) having serious difficulties in 
interpreting this. However, there were some suggestions that a glossary or some better 
explanation of terms would be useful, as would an explanation of the status and 
difference between ‘standards’ and ‘guidelines’ and a clearer demarcation between 
them; e.g. some of the guidelines currently appear to be interpreted or used as 
standards. 

Respondents also indicated that the ESG were relatively easy to apply in individual 
national contexts with 33 respondents saying that they were either very easy or quite 
easy to apply. Given the diversity of contexts within which agencies work across the 
EHEA, this could be seen as a measure of the success of the ESG to date. A question 
about the applicability of the ESG in an international context also yielded positive 
responses, although fewer agencies apply them in this way whereas all apply them in a 
national context. 

The concept of context is worthy of note: there were several discursive responses 
indicating that context-dependent application of the ESG was of significant importance: 
“ESG as it stands is just an outline - so it must be supplemented before it can be used by 
frontline agencies or institutions. This is a good thing because different systems at different 
stages of development will require different QA emphases and quality procedures need to 
evolve if they are to retain their vigour - a very detailed ESG could become an obstacle to 
such necessary variation.” 

Continuing the theme of context-dependent application of the ESG, 45% of 
respondents have difficulty in applying the ESG in their national context due to 
legislation (most of these problems lie in the areas of independence and resourcing), 
thus strengthening the view that long and detailed check-lists would not be useful. 

The questionnaire specifically asked agencies about their views on using the ESG as 
a check-list. There were many responses stating that it would be impossible to create 
an exhaustive list and the tension caused by the fact that the ESG form a major part 
of both the ENQA membership criteria and the criteria for inclusion on EQAR did 
not alter the strong disagreement (59%) with the notion of the ESG as a checklist. 

This opinion was reinforced by answers to a later question about whether or not the 
standards should be broken down into more detailed requirements for operational 
purposes - responses suggested that this would be too rigid, mechanistic and 
bureaucratic. 

When asked about potential difficulties in the application of the ESG by two bodies 
(ENQA and EQAR), 43% of respondents said that this was a problem. Comments 
included, “ENQA applies a principle-based approach (enhancement-led); EQAR a rules - 
based approach.” “If they consistently arrive at the same conclusions , what is the purpose 
of the double check? If they arrive at different conclusions , who will act as the final judge?” 

In terms of achieving consistency of application of the ESG, respondents showed 
a strong desire for developmental activity along the lines of exchange of information, 
sharing good practice, further training workshops and so on. Such suggestions could 
form the basis of a very profitable work-plan, not just for ENQA but for the E4 group 
in terms of working together to achieve mutual understanding and consistency across 


31 


the respective organisations and thus adding to the European dimension of quality 
assurance. 

Agencies were asked how easy/difficult/demanding the standards in each part of the 
ESG were to apply. Clearly from the responses received, agencies perceive themselves as 
having an important part to play in the application of all three parts of the ESG and not 
just parts two and three. 

Interestingly, in part one of the ESG, it was the same standard that was considered 
to be among the easiest, the most difficult and the most demanding for HEIs to apply 
(1.1 Policies and procedures for quality assurance.) One possible explanation for this is 
that the standard looks relatively simple on paper (i.e. in theory) but is more difficult to 
apply in practice. Another explanation is that respondents’ contexts mean that it is easy 
for some and difficult for others to apply. 

Part two of the ESG did not appear to cause agencies many problems in terms of 
applicability - there were more responses around which standards were easy to apply 
(mainly 2.7, periodic reviews) than those which were difficult or demanding (mainly 
2.3 Criteria for decisions). 

This trend follows into part three with significantly more respondents citing 
standards that were easy to apply (mainly 3.3 Activities) than those which were difficult 
or demanding (mainly 3.6 Independence). 

The standards and guidelines 

With regard to the idea of defining a threshold for partial/substantial compliance for 
individual standards, despite recognising that there are problems of definition for the 
terms partial’ and ‘substantial’ agencies still resisted the idea of a threshold, believing 
that it would create an unmanageable level of bureaucracy. (“There are too many 
national specificities. A threshold implies unfairness. It introduces a quantitative approach 
which is not fit for the purpose”) 

Following on from the point about needing further development of knowledge and 
use of the existing ESG rather than revision (see above), respondents acknowledged 
that clarification of certain aspects of the ESG would be necessary, not least the 
difference between ‘standards’ and ‘guidelines’ themselves. (“ Guidelines are interpreted 
as standards. This is perhaps the key problem of the ESG.” “Conceptually the difference 
is easy to grasp. The question is if and how the difference is consistently interpreted by 
agencies and panels.”) 

In response to a question about whether any of the standards were more central to 
the reviews of agencies than others, respondents were fully divided (24 said ‘yes’ and 23 
said ‘no’). 

• 73% agreed that the most central standard with regard to HEIs (Part 1) was 1.1 
(Policy and procedures for quality assurance). 

• 70% agreed that the most central standard for processes (part 2) was 2.4 
(Processes fit for purpose) and 

• 63% agreed that the most central standard for agencies (part 3) was 3.6 
(Independence). 

Whilst there is no agreement on the part of agencies in terms of the centrality of 
certain standards over others, this is not any great surprise given the responses to 
previous questions on the purpose of the ESG and the problems around definitions for 
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partial/substantial compliance. The standards cited as being possibly central are also 
not surprising. It is interesting to note that respondents were in stronger agreement 
about the standard for parts one and two but less so in terms of part three of the ESG. 

USE OF THE ESG IN THE EVALUATION OF AGENCIES 

Reponses to a cluster of questions around agencies’ experiences of being externally 
reviewed and the application of the ESG in those reviews revealed that the external 
review process is perceived to be valuable and important by agencies. External review is 
taken seriously for several reasons: 

• The review justifies and validates the work of agencies 

• The review process and the outcome (i.e. a report) provide agencies with a focus 
for enhancement 

• The ESG provide an opportunity for agencies to map their work against a set of 
internationally agreed principles. 

The notion of compliance with the ESG was not cited as one of the more beneficial 
parts of the review process. Responses included comments such as, “It provided the 
opportunity for the Agency to develop/refresh key corporate documents and gave external 
stimulus for the Agency to develop its own review methodology ...” and “It provided a 
catalyst for a raft of organisational and procedural changes which have benefited the 
agency greatly ...” 

However, respondents did note interpretation of some of the standards as being 
occasionally problematic during reviews. 

Revising the ESG 

At first sight respondents appeared to be divided on the need to revise the ESG. 
However, the majority opinion disagreed with revision at this stage: “Certainly there 
are elements which could be improved but none is so profound as to demand immediate 
revision. ..the general principles and standards are quite stable .” 

However, it is worth noting that those who felt that revision would be useful cited 
further clarification and explanation rather than actual revision in terms of content as 
the main reason for their response. In addition, suggestions for revision referred more 
to problems of overlaps and redundancies. A minority mentioned the inclusion of other 
Bologna action lines. 

Recommendations arising from the findings of the ENQA consultation and 
analysis of the findings by a group of experts 

The purpose of the meeting of ENQA experts was to ask that group to analyse and 
verify the findings of the consultation and to propose further recommendations if 
appropriate. The group carefully considered the outcomes of the consultation and 
offered a way forward for consideration by the E4 group. 

By and large, the expert group, having analysed the findings of the consultation, was 
able to verify its outcomes as presented in the main body of this report. 

Its analysis led the group to focus almost immediately on the matter highlighted in 
the section entitled Purpose and scope of the ESG above. It pinpointed the heart of any 
discussion around the revision of the ESG as being a question of whether one single 
document or reference point can fulfil the multiple purposes that have been assigned 
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to the ESG. The group agreed that the outcomes of the consultation indicated a plea for 
greater coherence in both the understanding and application of the ESG. The ESG have 
provided a means for the development of structures for quality assurance and a culture 
for quality. However, any revision would not be worthwhile if the intent and purpose of 
the document is not clarified. 

To that end, the group offered the following proposal, recognising that it may not be 
the right time for such action and that, even if the proposal was agreed, the timescale 
for such a revision to the ESG might necessarily be longer than is desirable at present 
(see below). 

The proposal suggests a set of overarching principles for quality assurance in the 
EHEA; these principles might be distilled from the current standards and would not 
necessarily be new. The principles would stand alone and would have the capacity to 
be used for various purposes. However, the principles themselves would not set out 
purposes or any means of achieving purposes - this would be entirely up to the body 
or organisation that wished to use the principles in a particular way. For example, 
ENQA might wish to use the principles as the core of its membership criteria. It would 
then be up to ENQA to decide whether it wanted to develop a further set of standards 
based on the principles that should be complied with, or whether it would prefer to 
work in the spirit of adherence to the principles themselves. Other organisations might 
choose to develop a different set of standards based on the principles, to use them for 
a different purpose to that of ENQA and to use them in a different way (e.g. as a set of 
accreditation criteria). 

Explanations (or guidelines’) for the principles (and for any standards developed by 
different organisations) would be set out in a separate document(s). 

As stated above, the ENQA working and expert groups recognise that, whilst this 
proposal offers a solution to the main problem (i.e. what is the purpose of the ESG?) 
it may not be currently feasible; we recognise the tensions between the analysis of the 
consultation and the possible consequences of following through the proposal on the 
one hand, and strategic/political deliberations on the other. 

ENQA therefore feels that it is important to set out, for the consideration of the rest 
of the MAP-ESG Steering Committee, those recommendations that may be drawn from 
the findings of the responses to the ENQA questionnaire prior to their analysis by the 
expert group. These may be summarised as follows: 

• There should be no immediate substantial revision to the ESG. However, in 
terms of short to medium term planning of work that might be carried out, some 
revision and clarification of the document might be beneficial 

• ENQA would propose revising and updating the "Introduction" and updating all 
contextual information in the document (this may mean removing some of the 
sections as the original document was a report rather than simply a reference 
point or set of standards and associated guidelines) 

• In the future (bearing in mind that the ESG are still being embedded in quality 
assurance policies and processes by higher education institutions and agencies, 
e.g. they have only been used in one full set of agency reviews so far), it might 
be useful to focus on individual standards and guidelines to ensure that there 

is no confusion or overlap and to ensure that there is consistency in terms of 
generality/specificity of each individual standard and its guidelines. It may also 
be interesting to consider providing examples in the guidelines of areas that 
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responses highlighted as currently missing from the ESG such as international 
collaboration and the impact of quality assurance processes on the quality of 
provision. (However, this might risk a list of inclusions that was not exhaustive - 
it may be preferable to consider strengthening any existing statements that declare 
that the ESG are applicable to all types of higher education no matter where or 
how it is provided or at what level). 

• A clarification of purpose or affirmation of the original purpose of the ESG would 
be invaluable. 

• The inclusion of a glossary would be useful in terms of clarifying terminology. 

Such clarification would include the terms used for categories of compliance. 

• Finally, this report also draws attention to the suggestion made earlier regarding 
the possibility of a shared work plan for the E4 group to continue to improve and 
develop knowledge and understanding of the ESG in order to ensure consistency 
of application by HEIs, review panels and their stakeholders. For example, ENQA 
now trains all external reviewers and publishes detailed guidelines on the review 
process. The training and the guidelines cover the use of the ESG. More sharing of 
the training process might help alleviate some problems such as the interpretation 
of some of the standards. 

NB. The majority of respondents (61%) felt that the members of the external review panel 
demonstrated a sound knowledge of the ESG and were well prepared for the review. 30% 
were mostly satisfied and 9% were unsatisfied. Given that many of these reviews would 
have taken place before the training for reviewers was formalised, this is an encouraging 
response. 

ENQA offers both the proposals of its expert group and the recommendations arising 
from the consultation for consideration by the other members of the E4 as represented 
on the MAP-ESG Steering Group in the spirit of ensuring that the future of the ESG is 
strong and has a clear purpose. 
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Annex 2: Executive Summary of the ESU 
consultation report 5 

This summary contains the key findings and recommendations. The full version of this 
report with more detailed information can be found online at www.esu-online.org 

The European Higher Education Area (EHEA) that is emerging would not become what 
it is aiming to be without a strong emphasis on quality and quality assurance, where the 
latter provides the basis for mutual trust - the prerequisite without which recognition, 
mobility and institutional cooperation can’t function on a larger scale. The European 
Standards and Guidelines (ESG) have been developed to cater for maintaining national 
characteristics of higher education and as a response to the essential need to create a 
mutual understanding of how to safeguard quality and to enhance it further. The ESG’s 
aim is to support the construction of the EHEA and ensure the trustworthiness of 
higher education to European society. 

The results of the analysis presented in this report underline the initial success of 
the ESG and the essential and ongoing need for them. At the same time, the findings 
and recommendations also provide concrete suggestions for further improvement of 
the ESG so that they could better contribute to the desired European Area of Quality 
Higher Education. 

The ESU consultation report of the MAP-ESG project builds on the experiences of 
national and local student organisations 6 , as well as individual student representatives 
and student reviewers who have been involved in quality assurance at institutional, 
national and European level. The consultation process was conducted through 
structured questionnaires, individual interviews and a focus group. Data was collected 
from 29 national student unions from 27 countries; 27 local student unions from 7 
countries; 11 individual students who took part in the external reviews of QA Agencies; 
and 21 focus group participants from 15 countries, who were also interviewed 
individually. The data analysis was complemented with text analysis of the ESG and 
relevant additional literature. 

Despite the variety of different data sources, several themes emerged which students 
identified as essential to a discussion on the usage and potential areas for enhancement 
of the ESG. These themes are Purpose and the scope of the ESG; Clarity and usability 
of the ESG; Impact and implementation of the ESG; and stakeholder involvement in 
quality assurance. A summary of key findings for each of the four identified themes is 
provided below, followed by a summary of recommendations for further improvement 
of the ESG. 

Purpose and scope of the ESG 

KEY FINDINGS 

• The ESG have been developed with a range of purposes and objectives in mind. 

• From the survey and focus groups it became obvious that students at both local 
and national level see the initially-stated purposes of the ESG, but have also 
identified other de facto purposes of ESG, and see both as important. This, in 


5 Authors: Bastian Baumann, Andrea Blattler, Nicolaas Heerens, Vanja Ivosevic, and Allan Pall 

6 Student unions and organisations at the institutional level 
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turn, poses the question of whether the ESG are suitable to fulfil these additional 
purposes, given that they might not have been designed for them. 

• Suitability of ESG with regards to its de facto purposes is particularly relevant 
regarding the use of the ESG as a reference checklist in assessing compliance, 
(which, it is often said, the ESG are not intended to be). Interestingly, the use of 
ESG as criteria was seen by national unions of students as the best outcome of 
the ESG. At the same time, the students attribute high relevance to the ESG to 
provide (the basis for) criteria to be used for reviews of programmes, HEIs and 
QA Agencies, as well as for making external QA more transparent and easier to 
understand. 

• With regards to further essential purposes of the ESG, in the view of students; 

(1) the ESG are an essential tool in forming a European Higher Education Area 
and in ensuring the provision of incentives for changes that would lead to the 
achievement of the goals agreed to in the Bologna Process and (2) the ESG are 
regarded as highly relevant for the establishment of a sound QA system, both 
nationally and at the level of higher education institutions. 

• However, the ESG live up neither to the expectations regarding greater 
transparency and easier understanding of external QA, nor to the idea that they 
improve education available to students. This is to be seen as one of the major 
shortcomings of the ESG, as the purposes themselves have been deemed very 
relevant and are often referred to in current debates on QA. But the lack of 
fulfilment of these purposes gives rise to questions on how greater transparency 
and a more direct effect on the improvement of education available to students 
could be achieved. 

• Students consider it essential to widen the scope of the ESG in order to better 
ensure that quality assurance contributes to high quality education. Attention 
needs to be paid to the necessary environment in which the development of the 
EHEA in real terms can take place and high quality higher education can prosper. 
For this, building trust between higher education systems and higher education 
institutions alike is essential. 

• Students view the necessity to widen the scope of the ESG in two ways. Firstly, 
within the current overall scope of the ESG, the standards and guidelines should 
be amended to include clearer linkages and references to the specific principles 
and commitments undertaken within the Bologna Process (this includes e.g. 
the social dimension, qualification frameworks and student-centered learning). 
Secondly, the consultation shows that further elements ought to be addressed by 
the ESG, in particular institutional management; research; lifelong learning; and 
transnational education. 

KEY RECOMMENDATION 

It is essential that a revision of the ESG is not limited only to minor amendments of the 
existing standards and guidelines, but involves a broader discussion on the purposes 
and scope of the ESG. This discussion on the purposes and scope of the ESG may be 
more demanding and challenging than a merely technical improvement of the current 
text of the ESG. The standards and guidelines form a holistic document which can only 
be meaningful when fully serving in the context of its overall purpose, which has seen 
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significant shifts taking into consideration the progress of the Bologna Process and the 
developments in the internationalisation of higher education. 

Clarity and usability of the ESG 

KEY FINDINGS 

• All the representatives of national student unions have indicated that they are 
aware of the ESG. More surprisingly, perhaps, is that on the level of HEIs, only 
a few student representatives indicate that they have not heard of the ESG. The 
majority of them claim to have some knowledge about the ESG, even though not 
always in detail. 

• The consultation identifies the language, phrasing and, in particular, the meaning 
of the specific quality assurance terminology used in the ESG as obstacles to the 
understanding and clarity of the Standards and Guidelines. 

• Students who have taken part in external reviews of quality assurance agencies 
assert that the external review panel members have high familiarity and 
understanding of the ESG. However, they highlight that the ESG also leave 
ample room for ambiguity. About half of the respondents indicated that there 
was little or no shared understanding between panel members in the external 
reviews where the ESG were used. Questions raised are touching on what the ESG 
mean, how they should be interpreted and how they should be used for reaching 
conclusions in reviews. 

• The focus groups and a careful text analysis of the ESG point clearly to the 
inconsistency of approach to individual standards and guidelines throughout the 
ESG. This inconsistency of the ESG is most visible in: 

- Significant differences in the precision of formulation of different standards 
and their accompanying guidelines. Whilst some are deemed rather precise 
and offer little room for interpretation, others are vague and use terms that are 
much more open to interpretation; 

- An inconsistent relationship between the standards and guidelines - there 
are several standards which are vague or not elaborated much, while their 
accompanying guidelines are very concrete and bring in important, additional 
elements for implementation and guidance. 

• Student representatives at all levels have noted that there is no clarity as to 
what the differences between standards and guidelines are supposed to be in 
their practical application. Students also reported the views of individual panel 
members differing significantly on this issue. In other words, if guidelines bring 
in a number of additional, important elements that are not addressed in the actual 
standard, to what extent can and should they be followed, or checked in external 
reviews. 

• The consultation thus shows that the ESG as a document lacks a coherent 
approach, resulting in inconsistencies in their application, which is 
counterproductive to the goal of enhancing trust and establishing transparency. 
Furthermore, there are indications that in practice standards are assigned 
different levels of priority, and are certainly prioritised over guidelines; it seems to 
be a matter of personal interpretation as to which standards are considered core 
and which less important. 
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KEY RECOMMENDATIONS 

In order for the ESG to achieve their purposes, it is essential not only to improve overall 
awareness and knowledge about them, but also for the ESG to be easily understandable 
to higher education stakeholders active at the European, and, most crucially, at the 
national and institutional level. 

The clarity of language and terminology could be enhanced through further 
elaborating the differences between national contexts and approaches to quality 
assurance in the introductory text to the ESG, where more up to date information and 
analysis of the different national QA systems might be underlined. 

The issue of ambiguity and achieving a shared understanding of the ESG may be 
more complex to address. However, it is essential, to enable consistent use of the 
ESG in external reviews of higher education institutions, study programmes and 
quality assurance agencies. Revising the ESG in a manner that will leave less room 
for interpretation might be a solution to this, although this may imply less room for 
flexibility. This clearly must be regarded as a trade-off which has to be discussed and for 
which a solution has to be found. 

Although not a solution to the lack of clarity of the text of the ESG itself, further 
suggestions for the improvement of the transparency and usability of the ESG could be, 
to include a glossary of terms in the context of the ESG purposes and a guide on how 
to use the ESG for external reviews and/or an explanatory report similar to that of the 
Lisbon Recognition Convention. 

To address the inconsistencies in the manner in which the standards and guidelines 
are written, a revision of the standards and guidelines as such is needed. The 
revision of the actual individual standards and guidelines should at least address the 
inconsistencies in the level of precision in which different standards and accompanying 
guidelines are formulated. 

Moreover, the relationship between the standards and guidelines should be 
discussed and clarified, particularly having in mind that some guidelines bring in new, 
additional elements to the standards. Thus, the role of the guidelines, particularly in 
the context of implementation and external reviews, should be clearly stated. 

Impact and implementation of the ESG 

KEY FINDINGS 

• The ESG have had considerable impact on the development of national QA 
systems since their adoption, either directly or as a guiding document. The use 
that is sometimes made of the ESG in the development and changes of quality 
assurance systems does, however, not imply that the standards and guidelines 
have been implemented consistently. The analysis of the data collected paints a 
very diverse picture of the way and extent to which standards and guidelines are 
actually being implemented within national higher education systems and higher 
education institutions. 

• Information received from the national unions of students suggests that all quality 
assurance systems have similar standards to those defined in the ESG as part 

of their systems, but at the same time these systems lack important elements of 
Standards or some Standards as a whole. 
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• A closer look at the institutional level shows that in all countries in our sample, 
as diverse as they are, there are HEIs with a system of internal QA in line with 
the ESG, but more HEIs which do not have this. The responses from local student 
representatives suggest that the implementation of a QA system that follows the 
principles of the ESG relies more on HEIs than on national governments. This 
means that it is possible that where in one institution certain Standards are 
almost completely translated into the internal QA system, another HEI in the next 
town might have a system in which a number of Standards and Guidelines are 
lacking. 

• With regards to student involvement, the textual analysis of the ESG suggests that 
regardless of referring to it as student involvement or student participation, or even 
student feedback, student engagement (yet another term often used) in QA can 
mean many different things to different people. Data obtained through the local 
student representative survey supports the impression that student participation 
is only relevant and encouraged in areas of QA where it is explicitly mentioned in 
the ESG. 

KEY RECOMMENDATION 

The consultation shows that despite a considerable impact of the ESG on national QA 
systems, there is a clear need for more consistent implementation of the ESG in internal 
quality assurance systems within higher education institutions. Building stronger links 
between external QA and internal QA, and therewith between parts 1, 2 and 3 of the 
ESG, seems appropriate. 

Student involvement should be defined more explicitly and consistently, both as a 
principle, and within different standards to fully integrate the principle of students as 
full members of the academic community, partners and active actors in university and 
higher education decision making processes, particularly those in QA. The Ministers’ 
summit in Bergen in 2005 provided initial guidance for defining student participation 
where the respective Stocktaking report already presented the student involvement in 
QA through specifying that it should include student participation in the governance of 
national bodies of QA, within teams for external review, consultation or involvement 
during external reviews and involvement in internal evaluations (Bologna Process 
Stocktaking, 2005, p.17). 

Furthermore, for students it is essential that their role is acknowledged in the 
decision making processes within higher education institutions and national education 
systems as full partners in the development of quality higher education. This means 
that students should be involved in the governance structures in higher education 
institutions and at the national level as far as higher education is concerned. In the 
follow-up of this E4 project mapping the ESG, students should be involved in the setting 
up of both national and institutional quality assurance systems and take active part in 
the quality assurance bodies created nationally and institutionally. 

Stakeholder involvement 

The ESG are a showcase for the unique way of cooperation that is a crucial factor 
for the success of the reforms in European higher education: the involvement of 
stakeholders. The ESG are the outcome of a wide cooperation; developed by higher 
education institutions, students and quality assurance agencies alike; adopted by 
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Ministers responsible for higher education. This has ensured their acceptance in the 
higher education community and beyond. 

Therefore, the discussion on the future of the ESG should be based on the same 
principle. The input provided through the consultations of each of the E4 partners 
provides an in-depth insight into how different stakeholders perceive the purpose and 
scope of the ESG and how they evaluate the ESG against these purposes, and give 
recommendations on how to further improve the ESG to best cater for them. 

As in any inclusive process, the views of different stakeholders sometimes differ and 
sometimes converge. It is in these views, especially those that are different, where the 
future of European cooperation in quality assurance is hidden. It is thus essential that 
in an upcoming revision of the ESG the relevant actors holding a stake are adequately 
involved to allow for the further improvement of the ESG and establishing the 
European Area of Quality Higher Education. This includes quality assurance agencies, 
higher education institutions, students as well as academic staff, governments, 
the European Quality Assurance Register and, where appropriate, labour market 
representatives. 
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Annex 3: EUA Consultation on the 
Implementation of the ESG 


Introduction 

Within the framework of the MAP-ESG project, EUA’s consultation among its members 
consisted of two parts: desk research and workshops. 

Desk research looked into existing information sources, such as the Trends 
2010 survey 7 and the results of a survey EUA had carried out in spring 2010 in the 
framework of the project Examining Quality Culture in Higher Education Institutions 8 
(EQC). The latter survey - with 222 HEIs responding from 38 countries - aimed at 
mapping existing quality assurance processes within HEIs which use the ESG as a 
reference framework. A summary of the key findings on the implementation of the ESG 
was included in the first project publication and is also attached to this report as an 
annex. 

As part of this desk research EUA also disseminated another concise survey to each 
of its collective members (the national rectors’ conferences from 34 countries). This 
survey, which was answered by 25 rectors’ conferences, focussed on the changes in the 
national or regional quality assurance frameworks and on the collective view on how 
the current state of play - with regard to external quality assurance in particular - 
reflects the principles underpinning the ESG. 

In addition, a series of workshops was organised offering the participating 
universities opportunities to benchmark their quality assurance processes in teaching 
and learning against the ESG, share their experiences of good practice as well as 
obstacles encountered in implementing these processes, and provide feedback on the 
ESG. 

The following three workshops were organised, each focusing on the key themes 
addressed by the ESG: 

• Policy and procedures for QA & information systems and public information 
(12-13 May 2011) 

• Approval, monitoring and periodical review of programmes and awards & 
assessment of students (17-18 May 2011) 

• QA of teaching staff & learning resources and student support (9-10 June 2011) 

Further to the specific themes of each workshop, the programme was designed to 
encourage the participants also to discuss the role of the ESG on a wider scale and 
to reflect on the other sections of the document. This allowed the organisers to 
triangulate the data collected. 

Following a call for interest, 49 delegates from 23 European countries participated in 
the workshops, thus contributing to a thorough discussion on how the ESG have been 
implemented and applied in institutional contexts. 

The key findings of the consultations and themes raised are discussed in the 
following section of this report. The detailed remarks made with regard to the actual 
text as presented in the ESG publication have not been included this report, but have 
been noted and will contribute to the future discussions on the ESG. 

7 Further information available in http://www.eua.be/eua-work-and-policy-area/building-the-european-higher-education-area/ 

trends-in-european-higher-education.aspx 

8 Further information available in http://www.eua.be/eua-work-and-policy-area/quality-assurance/projects/eqc/. 
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Analysis of findings 
Purpose and scope 

The discussions in the MAP-ESG workshops emphasised the importance of ESG as a 
reference framework for implementing a QA framework in most national systems and 
within most universities. The National Rectors' Conference (NRC) survey results also 
indicated that this is the way the ESG are generally understood at national level, with 
only one NRC out of 25 reporting that they are considered to be “requirements to be 
met” and the rest considering them as a reference framework. 

It is worth noting the observation made both by Trends 2010 and the EQC survey: 
the HEIs do not necessarily use the ESG directly as their source of reference when 
developing their internal QA, instead they “tend to show interest in one or several 
aspects of them” and often the principles of the ESG are embedded in the external QA 
requirements and thus are addressed by the HEIs as well (Loukkola & Zhang 2010: 35, 
see also Sursock & Smidt: 63). This was also confirmed by the university representatives 
in the MAP-ESG workshops. Most of them reported they were familiar with and had 
read the ESG, while there were a few - usually relatively new in the job - who admitted 
that they had read the document for the first time when preparing for the workshop. 
However, all of them felt that the principles presented in the ESG were familiar and 
relevant to them and addressed topics that their respective institutions were working 
on. The principles were often either already traditionally embedded in their existing 
institutional cultures or they had been introduced to the institutions through the 
requirements of the national QA systems or other regulations. 

These results indicate that the purpose of the ESG, at least when considering Part 1, 
is clear and corresponds to the original intentions of the authors. This finding is very 
different from that of QA agencies which have been discussing for a number of years 
now how the purpose of the ESG has changed from generic principles to becoming 
a compliance tool. Some further reflection on how the institutional representatives 
reflected upon this change is given below. 

Key messages from the EUA consultation in terms of purpose and scope of the ESG: 

• Maintaining the nature of generic principles is crucial. The experience shows 
that the ESG as they are formulated right now are adaptable to various political, 
legal and cultural settings as originally intended. The EUA consultations clearly 
demonstrated that they are applicable to institutions and agencies which function 
in different contexts and are of different sizes and orientation. It should be noted 
that this approach respects both the system level and institutional diversity, 
which is generally considered to be the strength of the European higher education 
landscape. 

• The recognition of institutional autonomy and responsibility for QA by 

the ESG is critical. The introductory chapters of the ESG identify the “central 
importance of the institutional autonomy, tempered by a recognition that 
this brings with it heavy responsibility” as one of the fundamental principles 
permeating the ESG (ENQA 2005: 10-11). On page 14, first among the basic 
principles of the ESG listed is for the HEIs to have the primary responsibility for 
quality assurance for what they offer. These principles were considered critical 
and are worth stressing even more in the future. 
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• While HEIs need to work on all-encompassing QA systems, the focus of the 
ESG should remain on the teaching and learning mission of the HEIs. 

While there are arguments in favour of including references to other missions of 
the HEIs - namely research and service to society - it was acknowledged that 
attempting to define joint principles for quality assurance processes in these fields 
through the ESG would probably not be feasible given the present responsibilities 
and modalities in place in national systems. This is even more important as 
developments in this direction would not necessarily gain acceptance from the 
parties concerned (i.e. first and foremost, the research community). 

• The purpose of the ESG should be to promote shared understanding of good 
practices in internal and external quality assurance that aim at quality 
improvement and increasing mutual trust among the stakeholders. This can 
only be reached by developing and promoting quality culture within HEIs and 
securing the commitment of all parties (institutional leadership, academic and 
other staff, students, quality assurance agencies, policy makers). In this context 
it is worth underlining that QA processes should not be expected to ensure 

the achievement of policy goals and implementation of policy tools unless they 
directly contribute to the aims identified above. 

• The link between research and teaching makes university education different 
from other forms of education and this link could be more clearly addressed in 
the ESG. An interesting observation is that while according to the introduction 
the document does cover all three cycles of higher education, it does not address 
matters that are crucial to research-oriented study programmes (either at Master 
or doctoral level). In this context EUA’s Salzburg Principles I and Salzburg 
Recommendation II were identified as main sources of reference. 9 

Implementation 

It is clear that there have been considerable changes in quality assurance since the 
adoption of the ESG, both at system level as well as at institutional level. Thus, it was 
not surprising that 63% of the HEIs responding to the Trends 2010 survey identified 
quality assurance reforms as being among the most important developments that had 
shaped their institutional strategy, QA being thus the second most important change 
after the Bologna Process (Sursock & Smidt 2010: 73). 

An ENQA study on external QA processes published in 2008 found that three- 
quarters of agencies responding to the survey had changed their quality assurance 
approach recently or were planning to do so (Costes & al 2008: 26). The EC report from 
2010 found that 11 countries had set up a quality assurance agency since 2005 (EC 
2010: 25). The survey of EUA’s collective members (NRC) further confirmed that there 
had indeed been changes in the national QA frameworks since 2005. 21 out of 25 NRCs 
reported that there had been changes in the national quality assurance system since 
2005. Whereas in a third of these cases (7) the changes were explicitly linked to the 
ESG, in half (11) the ESG were not directly mentioned, but they could be interpreted as 
underpinning the changes implemented. 

In terms of the developments in internal quality assurance, the EQC survey results 
showed that 52% of the responding institutions had introduced their institutional 
quality assurance system since 2005 or are currently designing it (Loukkola & Zhang 
2010: 21). The results also indicated that these QA systems clearly focus on the teaching 


9 Salzburg Principles from 2005 are available in http://www.eua.be/Libraries/CDE_website/Salzburg_Conclusions.sflb.ashx and 
Salzburg Recommendations II from 2010 in http://www.eua.be/Libraries/Publications_homepage_list/Salzburg_II_ 
Recommendations. sflb.ashx. 
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mission of the HEIs (Loukkola & Zhang 2010: 19). More detailed results from the EQC 
survey results in terms of ESG are presented in the annex of this report. 

Key message from the EUA consultation in terms of the implementation and the 
impact of the ESG is: 

• The results of previous studies combined with the findings of workshop 

discussions and the NRC survey leave us convinced that the development of the 
European framework for QA and the adoption of the ESG as a crucial part of it have 
had a major impact on how quality assurance in the European higher education is 
understood and carried out. 

Applicability 

Both the respondents to the NRC survey and the workshop participants found that 
the ESG were easy to apply. This was the case clearly because they considered the 
ESG to be a reference framework, and not simply a checklist. As many university 
representatives noted in the workshops, the principles presented by the ESG are 
widely accepted, but the real challenge - some might even call it difficulty - lies in 
the implementation: how to design and carry out processes that “comply” with these 
principles and are effective and supportive to the development of the HEIs. 

Both national and institutional quality assurance systems and processes form part 
of the larger national (or sometimes regional) HE systems and therefore cannot be 
completely taken out of the context in which they operate. In many systems, national/ 
regional authorities have - while taking account of the commitments made at European 
level - added special characteristics to both HE systems more generally and specifically 
to quality assurance arrangements in this context. 

However, discussions, in particular during spring 2011, demonstrate that unlike 
Part 1 of the ESG, there are more questions related to the applicability of the ESG 
Parts 2 and 3 and thus on external quality assurance. While even in this case many of 
the standards are not considered problematic and have made a positive contribution 
to the development of external QA processes, using the ESG as a compliance tool for 
QA agencies has raised some key issues related to external QA that do not seem to be 
addressed clearly enough in the document. 

The ESG do not explicitly define what “QA agencies” or “external QA processes” 
are. They do, however, acknowledge the diversity of QA purposes and approaches. 

This acknowledgement was considered important by the institutional representatives 
who saw it as being crucial to the further development of the European dimension in 
quality assurance and to EQAR, which uses the ESG as the criteria for inclusion on 
the Register, as it opens the European external QA market to different actors. This 
will ultimately, when the national legislation allows it, open the universities to the 
possibility of choosing the provider of external QA among different agencies. Thus, it 
was recommended that this acknowledgement of diversity of approaches and actors 
should be further stressed. 

Moreover, once the principle of using an extra-national agency is conceded by a 
government (under the EQAR establishment principle) the idea of requiring regular 
reviews becomes, in fact, irrelevant, as no national agency can enforce periodicity 
if HEIs are given a choice of agency. Even in the current situation many agencies 
cannot themselves enforce the periodicity, which is rather a question of national 
legislation. In this context, one is tempted to ask how relevant the standards 2.6 
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(follow-up procedures) and 2.7 (periodic reviews) are, as they should actually be seen 
as characteristics of the quality assurance system in which the agency works, rather 
than that of the agency itself. When judging the agencies under these criteria, the 
appropriate object of evaluation should in fact be whether it has the capacity to carry 
out follow-up procedures and whether it has pre-defined them in its guidelines and 
procedures. 

Key messages from the EUA consultation in terms of the applicability of the ESG: 

• Provided that the ESG are understood as a set of ‘ generic principles ” thus allowing 
cultural, regulatory and procedural diversity, they were found to be applicable and 
useful 

• The acknowledgement of diversity of approaches and actors in quality assurance is a 
crucial part of the European quality assurance framework and thus could be even 
further stressed in the ESG. 

Revising the ESG 

It is worth noting that the ESG are incorporated in various ways in national level 
policies and procedures, and changing the original document would require these 
policies and procedures to be revisited as well. Thus, a mere rewriting of the ESG would 
only be a first step and certainly not solve the unresolved challenges in the further 
development of the European Higher Education Area. 

During the workshops, when discussing a particular standard or guideline, some 
requests for further guidance on how to set up the processes or hands-on mentoring 
were made. However, there was widespread agreement that this should done through 
other means (such as workshops, nationally or regionally prepared guidelines, 
documents prepared to support peer learning), and cannot be achieved through high 
level policy documents such as the ESG. Furthermore, it was noted that often it is more 
natural to establish these platforms at national or at disciplinary level, for example 
through associations and informal networks. But it was also proposed that EUA - and 
other stakeholder organisations - should continue to offer their members opportunities 
for sharing experiences at grass-root level. 

It is interesting to note that most of the themes that gave rise to the liveliest 
discussions during the workshops related either to the recognition of the importance 
of the institutional quality cultures in assuring and enhancing quality or to the further 
developments in the EHEA since the adoption of the ESG, particularly to 

a. the changing purpose of the ESG, as they are now used as a compliance tool for 
quality assurance agencies or 

b. the development of thinking in terms of teaching (student-centred learning, 
diversity of student population and modes of delivery) or in terms of policy 
priorities (transparency). 

Key messages from the EUA consultation in terms of revising the ESG: 

• Considering the status of the ESG as a set of ‘generic principles’ for good practice 
in quality assurance in Europe, the conclusion is that while it might be useful to 
revisit the ESG, and reformulate some sections of the document, it is not necessary 
to change the underpinning principles. There is clear evidence that these are still 
both relevant for and of importance to the HEIs and agencies. 
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• Any revision should clarify the questions raised by the current ESG and make 
sure that they reflect the trends that have strengthened since 2005, namely 
the importance of transparency, student- centred learning, widening access, 
promotion of the European dimension in QA, also through EQAR, not to mention 
the importance of commitment of all stakeholders to promote the development of 
quality cultures within HEIs so that it genuinely contributes to the enhancement 
of quality. 

Conclusions 

In conclusion, the EUA consultation found that the generic principles defined in 
the ESG are still very valid and useful for HEIs and QA agencies as a framework 
within which to develop procedures to assure and enhance the quality of teaching 
provision. The importance of the respective standards or guidelines may vary from one 
institutional or national context to another, while shifts in national policy discussions 
may contribute to the changes in the priorities. 

While there is a consensus that some of the standards could certainly be clarified 
and better formulated, there also exists a consensus that any revision should not 
change the underlying principles, and thus the essence of the document. Moreover, 
some concerns were expressed that tinkering with the document here and there 
might, on the contrary, have negative consequences. Thus, should there be a revision, 
it should only be considered after careful deliberation and with the consensus of all 
major stakeholders so as to ensure continuity and that the work done so far is not 
undermined. 

However, another way forward was also proposed: rather than rewriting the current 
ESG to incorporate the above-mentioned developments, it was suggested that the 
revised document focus purely on principles, while other elements, including possible 
further explanations, would be presented and discussed in secondary explanatory 
documents. 

To conclude, we summarise the recommendations of the EUA consultation as 
follows: 

• Maintaining the ESG as a set of generic principles is crucial. 

• In line with the institutional autonomy, it is vital to recognise that the HEIs have 
the primary responsibility for the quality of their provision and quality assurance. 
This should remain the grounding principle of the further development of 
European quality assurance. 

• The purpose of the ESG should be to promote shared understanding of 
good practices in internal and external quality assurance that aim at quality 
improvement and increasing mutual trust among the stakeholders. 

• The focus of the ESG should remain on the teaching and learning mission of the 
HEIs. 

• The link between research and teaching makes university education different 
from other forms of education and this link could be more clearly addressed in the 
ESG. 

• The acknowledgement of the diversity of approaches and actors in quality 
assurance is a crucial part of the European quality assurance framework and thus 
could be even further stressed in the ESG. 


The ESG as a set of “generic principles” are still relevant and of importance to the 
HEIs and agencies and it is not necessary to change the underpinning principles. 
Any revision should clarify the questions raised by the current ESG and make 
sure that they reflect the trends that have strengthened since 2005, namely 
the importance of transparency, student-centred learning, widening access, 
promotion of the European dimension in QA, also through EQAR, not to mention 
the importance of commitment of all stakeholders to promote development of 
quality cultures within HEIs so that it genuinely contributes to the enhancement 
of quality. 
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Appendix to EUA report: Summary of 
the EQC findings on the implementation 
of the part 1 of the ESG 

Source Loukkola & Zhang 2010: 33 - 34 


CORRESPONDING ESG 

SUMMARY FINDINGS BASED ON THE SURVEY 

1.1 Policy and procedures lor quality 
assurance: 

Institutions should have a policy and associated 
procedures for the assurance of the quality and 
standards of their programmes and awards. They 
should also commit themselves explicitly to the 
development of a culture which recognises the 
importance of quality, and quality assurance, 
in their work. To achieve this, institutions 
should develop and implement a strategy for the 
continuous enhancement of quality. The strategy, 
policy and procedures should have a formal status 
and be publicly available. They should also include a 
role for students and other stakeholders. 

Two-thirds of the respondents had a separate 
institutional QA policy statement (67.1%) and in a 
quarter (24.8%) of the cases the quality statement was 
included in another institutional policy document. 
4.5% do not have any QA policy document. 

98.2% of the respondents answered that their quality 
assurance processes cover teaching and learning. 

1.2 Approval, monitoring and periodic review 
of programmes and awards: 

Institutions should have formal mechanisms for the 
approval, periodic review and monitoring of their 
programmes and awards. 

95.5% of the respondents have defined explicit learning 
outcomes for all or some of the study programmes and 
71.7% have made them publicly available. 

In most cases (85.1%) the curriculum is designed by a 
working group consisting of various stakeholders and 
ultimately approved at institutional level (41%) or by an 
external body (30.6%). 

Processes for reviewing and monitoring programmes 
vary greatly and most institutions use combinations of 
various processes. 

Involvement of stakeholders is not always transparent 
or structured. Students are involved in 40.5% of HEIs 
in measuring student workload and, when a working 
group prepares the curriculum, 50.8% of HEIs report 
that students are part of the group. 
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CORRESPONDING ESG 

SUMMARY FINDINGS BASED ON THE SURVEY 

1.3 Assessment of students: 

Students should be assessed using published 
criteria, regulations and procedures which are 
applied consistently. 

75.7% of institutions have clear, pre-defined 
examination or other assessment methods in place, 
including, for 66.7%, regulations covering student 
absence, illness or other mitigating circumstances. 
Most institutions have a mix of several features as 
mentioned in the guidelines. 

82.4% of institutions make the assessment methods 
and criteria publicly available through their website, 
study guides or equivalent. In about the same number 
of institutions, teachers inform the students about 
these methods and criteria at the beginning of the 
course. 

60.8% of institutions ensure that assessments are 
conducted securely in accordance with the institution’s 
stated procedures, and 48.2% of them have their 
administration checking that the procedures are 
followed. 

1.4 Quality assurance of teaching staff: 

Institutions should have ways of satisfying 
themselves that staff involved with the teaching of 
students are qualified and competent to do so. They 
should be available to those undertaking external 
reviews, and commented upon in reports. 

71.8% of institutions conduct student surveys, 

63.1% have specified their own requirements for 
competencies of permanent teaching staff when hiring 
them. 61.7% offer optional pedagogical training for 
teachers whilst 26.1% organise compulsory training. 
59% of institutions keep the information on teachers’ 
aptitudes and performance confidential and available 
only at the leadership level (institution and/or faculty 
and/or department). 

In the case of 22.3%, the legal framework does not 
foresee the possibility of removing an ineffective 
teacher. 

1.5 Learning resources and student support: 

Institutions should ensure that the resources 
available for the support of student learning are 
adequate and appropriate for each programme 
offered. 

Learning resources are quite commonly offered, the 
most common being library (93.2%) and computer 
services (90.1%). However, their regular monitoring 
and evaluation is not quite as common. 

1.6 Information systems: 

Institutions should ensure that they collect, analyse 
and use relevant information for the effective 
management of their programmes of study and 
other activities. 

93.2% of the institutions have centralised information 
systems in place that include information on their 
teaching mission. Most commonly this information 
includes: student progression and success rates (87.7%), 
profile of the student population (83.2%) and teacher- 
student ratio per faculty/department/institute (65.5%). 

1.7 Public information: 

Institutions should regularly publish up-to- 
date, impartial and objective information, both 
quantitative and qualitative, about the programmes 
and awards they are offering. 

The most commonly published information is on 
qualifications granted by the programme (86.9%), 
the teaching, learning and assessment procedures 
used within the programme (82.0%) and the intended 
learning outcomes of the programme (81.5%). All 
institutions offer, with a variety of features, some sort 
of information on their programmes. 
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EURASHE held a consultation on the impact and implementation of the European 
Standards and Guidelines for Quality Assurance in institutions and programmes with a 
professional orientation. 

We used an on-line questionnaire, had telephone and face-to-face interviews and 
organised workshops, in which not only our members were involved, but all those who 
had an affinity with our sector. 

We purposefully opted for different consultation methods as to guarantee an 
input from various sources (institutions, QA experts and stakeholders). Additionally, 
the proviso that the consultation would bear on the three distinct parts of the ESG 
attracted different categories of respondents (rectors, QA experts, students, alumni and 
sometimes employers). 

In a first mailing we addressed heads of institutions, who were asked to identify 
other respondents in their own environment, such as a QA coordinator, a student and 
an external’ stakeholder’ (not directly linked to their institution), such as employers, 
alumni, student counsellors, etc. They were asked to participate in an on-line survey 
on our web-site, containing a set of about 30 questions bearing on the three parts of 
the ESG. The deadline for participation was later extended to include all interested 
stakeholders in higher education, and when concluded we collected over 230 fully 
completed and Valid’ responses from 26 countries in the EHEA. 

The consultation in the form of workshops at a members’ event (the annual 
EURASHE conference, 1 April 2011, Nice) was originally targeted at the standard 
attendants at our members’ events (heads of institutions, programme and 
internationalisation directors), but the attendance proved to be much more diverse, 
and because of this the focus was nearly always on ‘knowledge’ and ‘awareness’ of the 
ESG on institutional level, rather than on the degree of implementation, although the 
process of the ESG implementation was also discussed. 

For this reason it was decided to add a third consultation method (next to the on-line 
survey and the workshop sessions), namely structured telephone and face-to-face 
interviews (called ‘in-depth interviews’ as the target group was this time a sample of 
those that had responded on-line).This selection of identified ‘interested persons’ in a 
number of countries were given targeted and ‘open’ questions which this time were 
geared more to the implementation and the ‘review’ perspective of the ESG. A set of 
detailed questions was then sent to the targeted persons, who had expressed their 
interest in such an in-depth questioning, proposing a prior written input from their 
part, and inviting them for a telephone/face-to-face interview of 30-60 minutes. 

For this part of the consultation an attempt was made to cover different geographical 
and cultural areas in the EHEA, based on the conclusions from the on-line consultation 
that the geographical and cultural spread was an important element when assessing the 
degree of both knowledge and implementation of the ESG. 


10 Authors: Guy Aelterman (ARTESIS University College, BE), Lucien Bollaert (NVAO, BE-NL), Stefan Delplace (EURASHE, BE), 
Caty Duykaerts (AEQES, BE), Dries Froyen (VLHORA, BE), Bryan Maguire (IADT, IRL), Barbara Michalk (HRK, D), Andreas 
Orphanides (EUC, CY), Janusz Pawlikowski (KRZaSP, PL), Iva Voldanova (EURASHE, BE) 
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The consultation involved countries having different QA systems and a varying 
experience with and knowhow of QA, due to an early or more recent involvement in the 
higher education reform process. 

The different approaches, using varying sources of consultation, resulted well- 
documented and extremely diverse findings, in which we have attempted to provide 
a comprehensive picture of the ESG implementation for the professional sector of the 
EHEA covered by EURASHE, within the given time constraints. 

The present report is a summative version of a larger EURASHE report on the 
implementation of the ESG in institutions of professional higher education, which will 
be made available in March 2012, together with the full questionnaire and a transcript 
of the main content of the telephone and face-to-face interviews. 

The structure of this summative report follows the agreed structure of the 
intended E4 report with the major findings derived from the consultation and a 
set of conclusions and recommendations based on these. In the findings, wherever 
that is possible, we indicated what the source of the information is (i.e. the different 
consultation methods, on-line survey, workshop sessions, in-depth interviews). 

The subtitles in the underlying report are the questions and statements from 
the on-line questionnaire, which is the basis of the EURASHE report on the 
implementation of the ESG. 

Awareness/knowledge of the ESG 

KNOWING EUROPEAN STANDARDS AND GUIDELINES FOR QUALITY 
ASSURANCE (ESG) 

More than two thirds of the respondents of the EURASHE survey claim to know the 
ESG. This is not surprising considering the profile of the respondents in the three 
consulted target groups (Nice workshops, online survey and in-depth interviews). 

HOW DID YOU BECOME ACQUAINTED WITH THE ESG? 

Most consulted persons came in contact with the ESG through the QA agencies 
(essentially in Western and Northern Europe). The QA agencies introduced the ESG in 
the institutions through External QA (EQA). The ESG standards are points of interest 
or are linked to the criteria used in EQA. Some respondents mentioned that they came 
in contact with the ESG as a panel member in EQA procedures. 

Others were aware of the ESG as they had been following the Bologna process since 
1999 and as a consequence knew the outcomes of the Bologna Ministerial meeting in 
Bergen in 2005. 

The EQAF meetings were mentioned several times as a reference in the 
dissemination of the ESG. A lot of people came for the first time in contact with the 
ESG in EQAF workshops or paper sessions. 

A final group of respondents, mostly coming from Eastern Europe, said they came in 
contact with the ESG through international projects (Tempus projects on QA or on the 
implementation of the Bologna Declaration priorities) or via networks of institutions 
and umbrella organisations. 

It is not surprising to see that most Northern and Western European institutions 
(including the Spanish ones) have known the ESG since 2005 and that they have since 
then included the standards in their External and Internal QA procedures. 
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Institutions in countries which joined the EHEA later, consequently implemented 
the Bologna Declaration priorities and the ESG at a later stage. It is not exceptional to 
see that some institutions or countries introduced the ESG in QA only very recently. 

BEING AWARE OF THE DIFFERENT CHAPTERS 

Generally spoken, a fair 75% of the respondents are familiar or know very well part 1 
of the ESG. Two thirds also state knowing part 2 quite well. Finally more than 50 % do 
not know part 3 at all, or very little of it. 

ELEMENTS PART OF INTERNAL QA 

Elements in Internal QA (IQA) strongly influenced by the ESG are the QA policy 
as such, the implementation of the policy into procedures and the periodicity of 
evaluations. It is interesting to see that more attention is paid to student assessments 
(an observation made in the online survey as well as in the in-depth interviews). 

Quality of staff, of learning resources and of student support are considered as 
less important in IQA. However these observations in the survey are somewhat in 
contradiction with other conclusions. 

Finally collecting data as a support for programme management or for public 
information is considered less important. 

INTERNAL QA REDESIGNED 

Most respondents mention that their institutions had an IQA system, and even an EQA 
system, years before the publication of the ESG. However under the influence of the QA 
agencies, most institutions changed their IQA, converging it to the ESG. The impact 
of the QA agencies on EQA as well as on IQA is considerable. It needs to be said that 
policy changes of the QA agencies and conversion towards ESG are strongly influenced 
or steered by governmental decisions, new legislation, international pressure and the 
fact that both ENQA and EQAR use the ESG as criteria. 

A lot of respondents mention that the IQA has become more consistent, that more 
attention is paid to students’ assessment, that external stakeholders are more involved 
and that the PDCA circle is now widely introduced in IQA. 

Students’ involvement increased considerably in internal and external QA. They 
participate in peer review panels, in programme and course surveys, formulate 
recommendations on QA to institutions, HE umbrella organisations and governments. 

As a consequence of the more cyclical character of external evaluations, attention for 
internal QA policy, processes and procedures increased automatically. Efficiency and 
workload control are important considerations. 

Multiplication effects, mutual learning and exchange of experiences are 
consequences of international and regional cooperation and networking within the 
Bologna process including QA development. 

Finally some respondents mention that the effect of the ESG is not limited to QA 
of education. The QA principles are often, directly or indirectly, applied in other 
institutional domains such as research, administration and management. 

INTERNAL QA CHECKED WITH ESG 

A direct check of IQA against the ESG is rare. Indeed the influence of ESG on IQA 
is rather indirect considering the requirements put by QA agencies in the EQA 
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procedures. Institutions know the ESG, especially parts 1 and 2. The ESG principles are 
implemented in the IQA but the link is rather implicit. 

NATIONAL QA PROVISIONS OR REGULATIONS CONNECTED TO THE ESG 

More than 50 % of the respondents agreed that national legislation and regulation on 
QA are redesigned under the influence of the ESG. Several authorities used the ‘stick’ of 
ESG for reorienting the national HE policy and for insisting on accountability from the 
HEIs. QA commitments (mandatory external evaluations, accreditations,. . .) and often 
a feeling of losing institutional autonomy are consequences of this process. However, 
most institutions welcome the QA changes and appreciate the stimulating effect of the 
ESG on the enhancement of institutional autonomy. 

Aims of the ESG 

HAVE THE OBJECTIVES BEEN ACHIEVED? 

When it comes to the question whether the objectives of the ESG have been achieved, 
the picture varies. Both HEI and stakeholders seem to have realised the impact of the 
ESG, but opinions differ, giving different impressions for internal and external QA. 

Almost two thirds of the respondents of the on-line survey are sceptical about the 
impact of the ESG on higher education in general, and on the fostering of “vibrant 
intellectual achievement” in particular. This mirrors the general question whether 
QA leads to a better learning experience for students. Probably the scope of the ESG is 
being viewed as a catalogue for QA criteria and procedures and the effect they can have 
on the intellectual life within an institution is only obvious for 35% of the respondents. 

This corresponds with the scepticism that the ESG fulfil their aim of informing and 
raising the expectations of HEI, students, employers and other stakeholders about the 
processes and outcomes of higher education. 60% of the respondents think that they 
have not done so or only in part. Again, in the minds of the respondents, the ESG are 
closely connected to QA as their predominant scope and objective. 

Both aims discussed here are not part of the core paragraphs of the ESG. To be aware 
of the fact that these objectives are inherent in the ESG, the reader has to read around 
the parts that give hands-on practical advice on what should be taken into account 
when working on an institution’s QA. So it can be concluded that if the ESG are not yet 
as effective in reaching these aims as they were expected to be, this may be the result 
of the reading patterns adopted: the parts of the text that are well perceived are those 
needed for daily work. 

This is confirmed by the fact that when asked whether the ESG succeeded in 
providing a source of assistance and guidance in QA, the number of negative answers 
was very low (3%). 97% of the respondents saw at least partly a success story here, half 
of them were convinced that the ESG had achieved their aim of being useful to HEIs 
and agencies when they worked on developing their own quality culture. Thus the ESG 
have established themselves as one of the main sources for HEIs and agencies when it 
comes to drawing up processes and procedures, criteria and culture. They have inspired 
developments and change in the institutions, agencies and organisations. 

This holds true especially for the agents of external QA. Half of the respondents were 
sure that the ESG have influenced the quality approach of quality assurance agencies. 

As two parts of the ESG out of three deal with the QA of QA agencies, and as the 


55 


compliance with the ESG is known to be crucial for the agencies to be awarded ENQA 
membership or to be listed on the EQAR, the ESG are clearly perceived as setting the 
standards for external QA and its actors. Those who think that this objective is only 
partly achieved are mainly living and working in countries in transition, where the 
agency has not yet gained enough independence from former structures and actors. 

Have the ESG achieved the aim of providing a “common framework”? As we will 
see later (cf. Implementation), this largely depends on a) when a country has taken up 
initiatives in QA in Higher Education (before 2005, after 2005) and b) when it joined 
the Bologna Process. 

HAVE THE STATED PURPOSES OF ESG BEEN REALISED? 

Only a third of the respondents are convinced of the direct impact of the ESG on the 
students’ learning experience, but another 61% state that they have “partly” improved 
the education. Thus, an overwhelming majority of more than 90% believe in a 
positive effect of the ESG on education across Europe. When it comes to the question 
of quality enhancement and, as a result of that, more institutional autonomy, almost 
half of the respondents see a close connection here, and the other half (only 6% deny 
the connection) attribute it partly to the ESG. This is a clear statement that QA, as a 
means of enhancement and accountability, has become a prerequisite for institutional 
autonomy, and is perceived as such by the HEIs. 

The ESG’s relevance for the work of QA agencies is firmly established: more than 
half of the respondents say that the ESG form the background for external QA. Another 
42% say that it is partly the case. We may assume that this is due to the different stage 
of QA-evolution that the countries, their HEIs and agencies respectively, are in. Added 
up, there is a 94% positive response versus only 6% of definite nays. The ESG are the 
accepted framework for external QA among the vast majority of respondents, and they 
are seen as an important “transparency tool”: only 8% of the respondents fail to see 
their contribution “to make external QA more transparent and simpler to understand” 
for everybody involved. 

Implementation 

When analysing the implementation of the ESG there are three sets of questions and 
answers that are looked upon: the usefulness of the ESG and the implementation of the 
ESG in both internal and external QA. 

a. Usefulness of the ESG 

USE OF THE ESG 

On the question how the ESG are actually implemented, a very large majority respond 
that the ESG are only used as inspirational principles or in broad compliance. This 
seems to be a healthy response, since the ESG were never meant to be used as criteria 
in a strict checklist, but as principles to be translated at the level of internal QA, 
external QA and QAAs. 

USEFULNESS OF THE ESG FOR THE INSTITUTION 

The same conclusion on implementation can be drawn from the questions about the 
usefulness of ESG for institutions. 
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42% of the respondents think that the ESG are rather theoretical, but can 
nonetheless help in the implementation of QA at institutional level. Once again the 
healthy mix of inspirational principles gains the majority. 35% of respondents think the 
ESG are a very practical tool. 

Looking at the geographical spread of the opinions on the practicality of the ESG, 
there is a possible link with the history of the development of a national QA system. In 
countries where a national QA system was developed before 2005, as was the case in 
the Netherlands and Finland, the ESG are regarded as a more theoretical framework. 
While in countries where a national QA system was developed after 2005, like in 
Bulgaria and Kazakhstan, the ESG are regarded more as a practical tool. It is known 
that in the latter countries the ESG were more than inspirational to the development of 
a national external QA system. Thus the ESG entered into internal QA via external QA. 
Yet this has never been the concept nor the aim of the ESG. The in-depth interviews 
confirm this historical dimension. 

USEFULNESS OF THE ESG FOR QAAS V. INSTITUTIONS 

This finding is confirmed by the (different national) responses on the usefulness of 
the ESG at institutional level as compared to QAAs. A large majority (69%) think that 
the ESG are equally useful for both, while 25% think that the ESG are more useful for 
QAAs and only 5% think that they are more useful for the institution. 

USEFULNESS OF THE ESG TO THE DIFFERENT ACTORS IN THE 
INSTITUTION 

As for the question of the usefulness of the ESG for the different institutional actors, a 
large majority (61%) experience that the ESG are most useful to the QA management. 

It is a positive observation that the second category to whom ESG are most useful is 
identified as the Board of Directors/the Management. This means that the leadership 
of the institutions take their responsibility for QA. On the other hand the fact that the 
usefulness for academics or teaching staff is only considered by 12%, which is lower 
than for the administrative staff (16%), is an indication that QA is still regarded as top- 
down and as an administrative task rather than the heart of the matter for academics. 
The fact that the ESG were approved in 2005 and first implemented in the national 
development of external QA and for the registration of QAAs by EQAR from 2008 
onwards, which, in many countries, was before a healthy and participative internal 
QA system was established at institutional level, certainly plays a role here. The fact 
that the ESG usefulness to other traditional stakeholders of HE (students, alumni 
and employers) is only rated from 10% to 3%, is an indication that the ESG are not 
specifically focusing on them (although the ESG strongly advocate the stakeholders’ 
model and the participation of students) and that these stakeholders are not (yet) very 
much involved in QA either. 

SATISFACTION WITH THE IMPLEMENTATION OF THE ESG BY THE QAAS 

The fact that 70% of all respondents are rather satisfied with the implementation 
of the ESG by QAAs proves that most know the ESG relationship with external QA 
through QAAs and 70% think that the relationship is healthy, and at the same time are 
conscious of some tensions. The low percentages of both extremes (very satisfied 12% 
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and not satisfied at all 2%) prove that the relationship between internal and external 
QA is not one to one. 

b. Implementation of the ESG in internal QA 

Although former surveys on institutions and internal QA seem to indicate that they 
were less aware of the ESG and used the ESG not or far less at institutional level, the 
EURASHE survey seems to contradict this. Not only does the awareness at institutional 
level seem high, but also the implementation of the different aspects of the ESG in 
internal QA appears to be very high. It would be interesting to investigate whether 
this difference in degree of knowledge and implementation is dependent on the sector 
(university - non-university HE) that is surveyed, on the time of implementation (later 
in the non-university sector) or influenced by both factors. 

ELEMENTS CONNECTED TO THE INTERNAL QA OF THE INSTITUTION 

The fact that policy for QA is rated highest in terms of implementation of the elements 
of QA in the institutions, may not be a surprise, since the ESG are a Bologna document 
and are seen by many as a political decision or certainly influenced by the national, 
and thus political, implementation. The fact that in some countries and certainly in 
some institutions the development of internal QA has only recently started or is (being) 
redesigned adds to this perception of being primarily a policy for QA’. From the analysis 
of the general use of the ESG it has already become clear that internal QA is primarily 
seen as a matter for the executives and QA (administrative) management. 

The fact that the lowest score of ESG implementation is given to information and 
public availability of QA results may not come as a surprise. In quite some countries 
where the national external QA has only recently been developed or changed, no 
reports have been published yet. Even in countries where an external QA has been in 
place for long, the quality culture has not yet reached a stage where information about 
programmes and awards from internal QA are made public. Self-evaluation reports are 
still seldom public and in most cases only used for internal aims and/or in preparation 
of external site-visit and reports. The fact that still 74% of our respondents give a high 
rating to publicly available information may refer to information in course guides, on 
curricula and degrees that is available on the institutions’ websites or in folders, but 
which is not a result of internal QA. 

c. Implementation of the ESG in external QA 

CONNECTION OF THE NATIONAL QA PROVISIONS AND REGULATIONS TO 
THE ESG 

While the way of implementing the ESG in internal QA seems to be characterised by 
a healthy mix of inspirational principles, the national QA provisions seem to be more 
designed on the structure of the ESG. Also this may not be surprising. It has been a 
general observation that the ESG as adopted by the Bologna ministers in 2005 had 
their first influence on the development of national external QA systems and the role 
of QAAs in these. The fact that both ENQA and EQAR, each in their own way, use the 
ESG as criteria for membership or registration of QAAs has had a fairly strong impact 
on both the functioning of QAAs and the national external systems. The national QA 
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provisions that were only inspired by the ESG represent only a bit more than half of 
those definitely designed on the ESG structures. Although the ESG took existing (best) 
practices into consideration while being written, the most strictly ESG-based national 
systems are those developed or redesigned after 2005. 

Appropriateness 

DEGREE OF SUPPORT OF THE ESG FOR QA 

The ESG have proven to be a huge success when it comes to supporting the 
characteristics of QA, such as harmonisation of principles and criteria, quality 
improvement and culture and the improvement of quality procedures. Overall, more 
than 95% of the respondents are of this opinion. 

The strongest support among the topics mentioned above goes to the improvement 
of quality assurance procedures. This shows that the ESG - although giving HEIs and 
agencies a wide range of methods by not defining any - have had an enormous impact 
on what is going on in QA in the EHEA. They are appropriate for supporting the HEIs 
in the task of QA as the evidence shows. 

When having a closer look at some of the topics of quality culture and improvement, 
the respondents are overwhelmingly convinced of the appropriateness of the ESG: in 
both cases the results are almost identical, as 98% think that the ESG support quality 
culture and improvement in varying degrees (support, support strongly or slightly). 

This is proof that the ESG, as they were written some years ago, are building a very 
strong framework for QA. Their standards and their guidelines indeed have the effect 
of harmonising QA. This is even more impressing as they - as stated above - have this 
impact without prescribing ways, procedures and methods. They have succeeded in 
improving and enhancing QA without being a detailed rulebook with boxes to tick. To 
quote Peter Williams’ speech at the 5th EQAF, QA in Europe is indeed “One fleet, many 
ships, same destination”. 

At one point the wine is watered down a bit: when it comes to information to the 
public and the support for fixed reference points, the “strong support” goes down 
to 15%. We have already seen in the part on the "Aims of the ESG", that the aim 
of improving information seems to be the one that is in want. On the other hand, 
the publication of complete evaluation or accreditation reports often means a more 
challenging move for HEIs or agencies than quality development as such. In many 
cases, there is no tradition of public information that contains not only “good news” on 
an institution. Regarding the fact that the HEIs are competing for the best students, 
this is not surprising, and even the ESG have not yet been able to change this behaviour 
all over the EHEA-countries. Nevertheless, an additional 50% feel that the ESG support 
information to the public, so we should take this as “work in progress”. 

Regarding the feature of supporting fixed reference points for the institutions, the 
ESG never aimed at providing them. We have already seen that they have fulfilled 
their task of being a framework, a fence around QA, and were never meant to be 
a prescriptive list of features to be checked. If they, nevertheless, get acclaim for 
supporting HEIs in developing QA reference points from 72% of the respondents (95%, 
if we add those who detect a slight support), this is indeed an indicator for “exceeding 
expectations”. 
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In the opinion of the survey respondents the greatest overall support is for the QA 
system itself, and as is generally felt, the ESG are found the least supportive in their 
informative role towards the public. Though they may never have meant to have such 
an explicit role, the assumption that this informative role is embedded in the present 
QA structures is clearly disproven here. Their support for quality culture and quality 
improvement is also recognised. 

RATING THE ESG ON A SET OF OBJECTIVE CRITERIA? 

The respondents were asked to give an opinion on whether the clarity of the ESG is 
appropriate. When looking at the results, we have to keep in mind that the ESG were 
drafted by a group of people in which the native speakers were a minority. So is the 
largest part of their readers: for most of them English is a second, if not a third or 
fourth language. 

Even with that barrier, 80-90% of the readers feel that the ESG are clear on a very 
high or rather high level. In several discussions it has been suggested that a glossary 
added to the ESG would make them easier to understand, but we all have faced the 
difficulties of defining concepts that have different traditions in different countries and 
are traditionally perceived in different ways. In the most important area of conceptual 
clarity, the ESG have scored the highest approval: this means that the message the ESG 
transport gets across to the reader and user. One should keep that in mind when calling 
for a radical reform of the text. 

The level of detail is rated worst in this question, but it is questionable whether this 
result should be heeded. Providing a detailed description and being a “how-to”-book has 
never been the aim of the ESG. It speaks for the readers that most of them (80 %) do 
not call for greater detail, having grasped the spirit of the ESG. 

Review section 

To the straight question “Should the ESG be revised?” the majority of the respondents 
answered positively (73%). To the question “How should the ESG be revised?”, the 
same percentage (73%) stated that the ESG should be modified slightly; 26% of the 
respondents said that the ESG should be modified intensively, while one per cent 
suggested starting from scratch. 

Once this assumed, next comes the issue of considering how to cope with a possible 
revision, what modifications to bring, with what impact on which actors, and finally 
how to proceed to carry on this possible revision process. 

If they suggested modification for the ESG, the respondents were submitted five 
possible answers. Here is what should be changed, according to them, from the highest 
to the lowest score: 

• 87% of the respondents would like to have the level of awareness among the 
general public changed/raised 

• 75% of the respondents agreed with the addition of new elements 

• 66% of the respondents would like to have the distinction between standards and 
guidelines modified (but without further specification) 

• A minority of 32% of the respondents consider that the structure in three parts 
needs modification 

• Only 31% of the respondents would see “must” replace “should” in the ESG. 
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A low level of awareness among the general public was one of the striking findings of 

the workshops organised by EURASHE. 

Conclusions and recommendations from EURASHE 

AWARENESS AND KNOWLEDGE OF THE ESG 

• Two thirds of the respondents claim they ‘know’ the ESG, but the degree of 
knowledge varies greatly. 

• Respondents know the ESG mainly through the external quality assurance (EQA), 
and especially in Northern and Western Europe the QAAs are given as the main 
source. 

• The different parts of the ESG are known by over 75 % (part 1), 66 % (part 2) and 
less than 50 % (part 3). 

AIMS OF THE ESG 

• Where the ESG seem to be most effective is when they give hands-on practical 
advice on what should be taken into account when working on an institution’s QA. 

• Whether the ESG achieved the aim of providing a “common framework” for QA, 
largely depends on a) when a country has taken up initiatives in QA in Higher 
Education (before 2005, after 2005) and b) when it joined the Bologna Process. 
Where this happened late, the countries may still be on the way to modelling their 
QA systems according to the ESG, thus being still in transition and the ESG are 
not perceived as a strong framework. 

IMPLEMENTATION OF THE ESG 

Usefulness of the ESG 

There are important ‘regional’ differences in the state of implementation of the ESG in 

the EHEA. 

• A large majority of the reviewed people say that the ESG are only used as 
‘inspirational principles’ or ’broad compliance’, rather than as a ‘checklist’. 

• There may be a link between the history of the development of a national QA 
system and the ‘use’ that is made of the ESG in a particular country. In countries 
where a national QA system was developed before 2005, the ESG are regarded 
more as a theoretical framework, whereas in countries where this happened later, 
the ESG are regarded more as a practical tool. 

• The countries/regions rate the ‘practical usefulness’ of the ESG very differently, 
with the ‘new Bologna countries’ rating it much higher than those who have had 
an advanced quality system for a long time, which was brought in line with the 
ESG quite early. 

• When it comes to defining the ‘actors’ at institutional level who take the greatest 
benefit from the ESG, the Board and the QA managers come with a significant 
distance before the other categories (administrators, academic staff, students and 
alumni). 

• In the opinion of the respondents the greatest overall support is for the QA system 
itself, and the ESG are found the least supportive in their informative role towards 
the public. Though they may never have meant to have such an explicit role, the 
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assumption that this informative role is embedded in the present QA structures, is 
clearly disproven here. Their support for quality culture and quality improvement 
is also recognised in the on-line survey and the interviews. 

• Respondents to the survey are by a large majority (70 %) rather satisfied’ 
about the implementation of the ESG by the QAA, but very few express a ‘high 
appreciation rate’ (16 %). Judged per country, we see once again a more positive 
attitude towards the ESG coming from the ‘new Bologna’ countries. 

IMPLEMENTATION OF THE ESG IN IQA 

• The ESG have contributed most to the implementation of a quality assurance 
policy in an institution. 

• From the analysis of the general use of ESG it has already become clear 
that internal QA is primarily seen as a matter for the executives and QA 
(administrative) management. 

IMPLEMENTATION OF THE ESG IN EQA 

• The national QA provisions that were ‘only inspired’ by the ESG represent only 
slightly more than half of those ‘definitely designed’ on the ESG structures. 
Although the ESG took existing (best) practices into consideration while being 
written, the most strictly ESG-based national systems are the ones developed or 
redesigned after 2005. 

Appropriateness 

The ESG never aimed at becoming a framework for institutions, so if they get acclaim 
for supporting HEIs in developing QA reference points, this is indeed an indicator for 
‘exceeding expectations’. 

In most countries there is no tradition of ‘public information’ on the results of QA 
proceedings, so as to the role of the ESG in supporting information to the public, we 
should take it as “work in progress”. 

According to our findings on their ‘appropriateness’ the ESG get the highest rating 
for "Conceptual clarity", which means that the message carried by the ESG gets across 
to the reader and user. This should be kept in mind when calling for a radical reform of 
the text. 

Other features such as "Level of detail", "Linguistic clarity", "Internal consistency" 
and "Balance between the different parts" equally receive a relatively low rating. 

Impact of the ESG 

• The standards and guidelines have had the effect of harmonising QA. They have 
succeeded in improving and enhancing QA without being a detailed rulebook 
with boxes to tick. 

• Most institutions changed their IQA, converging it to the ESG standards, under 
the influence of QA agencies. Generally, the impact of the QA agencies on EQA as 
well as on IQA is considerable. 

• Students’ involvement has increased considerably in both internal and external 
QA. 
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• Some respondents mention that the effect of the ESG is not limited to QA of 
education. The QA principles are often, directly or indirectly, applied in other 
institutional domains such as research, administration and management. 

• Most institutions know the ESG, especially parts 1 and 2, via EQA. The ESG 
principles are also implemented in the IQA but the link is rather implicit. 

Most institutions welcome the changes in QA and appreciate the stimulating effect of 
the ESG on the enhancement of institutional autonomy. 

Review 

• A significant number of respondents think that the ESG should be revised, but the 
majority think that there can only be slight revisions, in the form of an update. 

• In some countries the implementation of internal and external QA proceedings in 
the non-university sector’ (and as a consequence also of the ESG) may have come 
later than was the case in universities, which means that a full revision of the ESG 
at this moment would be untimely. This impression is confirmed more precisely 
in the in-depth interviews, which reveal that some institutions have only recently 
started implementing the ESG. 

• There is no doubt that the implementation of the ESG has a beneficial effect on 
other activities of institutions than teaching and learning, notably in the way 
research is managed in the institution, and also in its service to the community. 

• The link between the ESG and the other instruments and realisations of the 
Bologna Process (of which they are an output themselves) are acknowledged by 
most respondents. However, how they relate to each other, is confusing to most of 
the respondents and therefore setting the ESG in the overall Bologna framework 
is required, which could be in the introductory text of the ESG document. 
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Annex 5: Abbreviations 


BFUG 

ECTS 

EHEA 

ENQA 

EQAR 

ESG 

ESU 

EURASHE 

E4 

HE 

HEI 

QA 


Bologna Follow-up Group 

European Credit Transfer and Accumulation System 
European Higher Education Area 

European Association for Quality Assurance in Higher Education 

European Quality Assurance Register for Higher Education 

European Standards and Guidelines, or in full. Standards and Guidelines 
for Quality Assurance in the European Higher Education Area 

European Students’ Union 

European Association of Institutions in Higher Education 

partnership between ENQA, ESU, EUA and EURASHE, 
related to quality assurance 

Higher Education 

Higher Education Institution 

Quality Assurance 

Quality Assurance Agency 


QAA 
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analysis of expert views from all relevant actors and stakeholders in higher education, notably teachers and students, 
higher education management and internal quality assurance bodies, external quality assurance agencies, EQAR and 
ministries, and the social partners. It is the first overarching study on the implementation of the ESG in all its parts, and a 
thorough analysis of their impact on European higher education. 
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